How we improve loading speed from 20-30 seconds to instant
This year, MetricsWave is growing a lot. The number of events has 3x in the last months, and we still run in a small server. Even that the service still has 99.99% uptime, our dashboards performance decreased and the experience was not good enough, until today.
A bit of context
The main problem was related to the visit metrics and the data associated with them (Country, Language, Reference, Destination Page, etc).
As of today we have +21 million different records. It's not a huge number, it's a manageable number, but this project is still running on a small, simple server and that was a challenge.
Each query to this data took, on average, between 10-20 seconds. This was clearly reflected in the dashboards, affecting loading time.
Furthermore, this data had become unmanageable. Making a transformation on the data was totally unfeasible.
What have we done?
Many changes, really.
We have realized that most users access recent data and rarely the history of the previous year, for example.
Also, in most cases, in the dashboard, each widget shows a single piece of data, but we had the loading optimized to obtain all the data at the same time.
And finally, many times we access exactly the same data, but simply changing the display of it.
With all this information we have made several changes to the structure in which we store the data and how we now load it dashboards load instantly.
We are preparing a more technical article about the exact changes we did on the database and queries. If you are interested, take a look at developerjoy.co.