Most of the models we build use quite a lot of data, and especially time series or time-stamped data. However, many of them don’t use huge, enormous petabytes of data. A data lake, not a data ocean. Handling this can be a bit tricky – spreadsheets are much too limited, and relational databases are generally too slow to read and (especially) write even moderate volumes of time series data. At the other end of the scale, large no-sql data solutions are just too complex and unwieldy, like trying to crack a nut with a hammer.
We needed something that could read and write time series and time-stamped data quickly; could classify, group and tag series (“inflow series”, “cases” or “scenarios”); could extract series even with missing or incomplete data; and had an overhead low enough that it could be installed on a desktop and accessed via the python tools we use every day.
We couldn’t find it, so we decided to build our own – the Puma Analytic Framework, for the flexible and rapid storage, retrieval, visualisation and manipulation of time series and time-stamped data. Implemented as a python package, the Framework uses a combination of a relational database and fast flat file storage and retrieval to create a tool to easily and cheaply store pretty large data volumes on the desktop, and use them in your data analysis and models. Being a python package, you have immediate access to all your usual python tools, such as numpy, scipy, pandas and Jupyter Notebooks.
The Framework makes developing new models and analytic tools easier and faster, by providing data in a readily accessible, standardised format. We use the Framework to store and prepare data for our own PUMA Algorithms power market model, and in the same way it can be fully integrated with your own existing and 3rd party power market modelling tools. This is one of the big advantages of the PUMA Analytic Framework in our opinion – it enables you to have a single, consistent database for all your models.
The PUMA Analytic Framework is in locked alpha testing and will be made available in Beta shortly, after which you will be able to download the package and documentation from our website for free. Email us in the meantime if you are interested in finding out more.