.. _tut_install_load_data: Toy example: Introduction and setup =================================== This page is a starting point of a series of tutorials that will help you get practical experience with FlexMeasures. Let's walk through an example from scratch! We'll ... - install FlexMeasures - create an account - load hourly prices What do you need? Your own computer, with one of two situations: either you have `Docker `_ or your computer supports Python 3.9+, pip and PostgresDB. The former might be easier, see the installation step below. But you choose. Below are the ``flexmeasures`` CLI commands we'll run, and which we'll explain step by step. There are some other crucial steps for installation and setup, so this becomes a complete example from scratch, but this is the meat: .. code-block:: bash # setup an account with a user, assets for battery & solar and an energy market (ID 1) $ flexmeasures add toy-account # load prices to optimize schedules against $ flexmeasures add beliefs --sensor 1 --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam Okay, let's get started! .. note:: You can copy the commands by hovering on the top right corner of code examples. You'll copy only the commands, not the output! Install Flexmeasures and the database --------------------------------------- .. tabs:: .. tab:: Docker If `docker `_ is running on your system, you're good to go. Otherwise, see `here `_. We start by installing the FlexMeasures platform, and then use Docker to run a postgres database and tell FlexMeasures to create all tables. .. code-block:: bash $ docker pull lfenergy/flexmeasures:latest $ docker pull postgres $ docker network create flexmeasures_network .. note:: A tip on Linux/macOS ― You might have the ``docker`` command, but need `sudo` rights to execute it. ``alias docker='sudo docker'`` enables you to still run this tutorial. After running these commands, we can start the Postgres database server (with user `postgres` and a new database `flexmeasures_db`) and the FlexMeasures server. That is done with the following commands: .. code-block:: bash $ docker run --rm --name flexmeasures-tutorial-db -e POSTGRES_PASSWORD=fm-db-passwd -e POSTGRES_DB=flexmeasures-db -d --network=flexmeasures_network postgres:latest $ docker run --rm --name flexmeasures-tutorial-fm --env SQLALCHEMY_DATABASE_URI=postgresql://postgres:fm-db-passwd@flexmeasures-tutorial-db:5432/flexmeasures-db --env SECRET_KEY=notsecret --env SECURITY_TOTP_SECRETS='{"1": "something-secret"}' --env FLEXMEASURES_ENV=development --env LOGGING_LEVEL=INFO -d --network=flexmeasures_network -p 5000:5000 lfenergy/flexmeasures When the app has started, the FlexMeasures UI should be available at http://localhost:5000 in your browser. .. include:: ../notes/macOS-docker-port-note.rst To establish the FlexMeasures database structure, execute: .. code-block:: bash $ docker exec flexmeasures-tutorial-fm bash -c "flexmeasures db upgrade" Now - what's *very important* to remember is this: The rest of this tutorial will happen *inside* the ``flexmeasures-tutorial-fm`` container! This is how you hop inside the container and run a terminal there: .. code-block:: bash $ docker exec -it flexmeasures-tutorial-fm bash To leave the container session, hold CTRL-D or type "exit". To stop the containers, you can type .. code-block:: bash $ docker stop flexmeasures-tutorial-db $ docker stop flexmeasures-tutorial-fm To start the containers again, do this (note that re-running the `docker run` commands above *deletes and re-creates* all data!): .. code-block:: bash $ docker start flexmeasures-tutorial-db $ docker start flexmeasures-tutorial-fm .. note:: Got docker-compose? You could run this tutorial with 5 containers :) ― Go to :ref:`docker-compose-tutorial`. .. tab:: On your PC This example is from scratch, so we'll assume you have nothing prepared but a (Unix) computer with Python (3.9+) and two well-known developer tools, `pip `_ and `postgres `_. We'll create a database for FlexMeasures: .. code-block:: bash $ sudo -i -u postgres $ createdb -U postgres flexmeasures-db $ createuser --pwprompt -U postgres flexmeasures-user # enter your password, we'll use "fm-db-passwd" $ exit Then, we can install FlexMeasures itself, set some variables and tell FlexMeasures to create all tables: .. code-block:: bash $ pip install flexmeasures $ pip install highspy $ export SQLALCHEMY_DATABASE_URI="postgresql://flexmeasures-user:fm-db-passwd@localhost:5432/flexmeasures-db" SECRET_KEY=notsecret SECURITY_TOTP_SECRETS={"1": "notsecret"} LOGGING_LEVEL="INFO" DEBUG=0 $ export FLEXMEASURES_ENV="development" $ flexmeasures db upgrade .. note:: When installing with ``pip``, on some platforms problems might come up (e.g. macOS, Windows). One reason is that FlexMeasures requires some libraries with lots of C code support (e.g. Numpy). One way out is to use Docker, which uses a prepared Linux image, so it'll definitely work. In case you want to re-run the tutorial, then it's recommended to delete the old database and create a fresh one. Run the following command to create a clean database with a new user, where it is optional. If you don't provide the user, then the default `postgres` user will be used to create the database. .. code-block:: bash $ make clean-db db_name=flexmeasures-db [db_user=flexmeasures] To start the web application, you can run: .. code-block:: bash $ flexmeasures run When started, the FlexMeasures UI should be available at http://localhost:5000 in your browser. .. include:: ../notes/macOS-port-note.rst .. _tut_load_data: Add some structural data --------------------------------------- The data we need for our example is both structural (e.g. a company account, a user, an asset) and numeric (we want market prices to optimize against). Let's create the structural data first. FlexMeasures offers a command to create a toy account with a battery: .. code-block:: bash $ flexmeasures add toy-account --kind battery Generic asset type `solar` created successfully. Generic asset type `wind` created successfully. Generic asset type `one-way_evse` created successfully. Generic asset type `two-way_evse` created successfully. Generic asset type `battery` created successfully. Generic asset type `building` created successfully. Generic asset type `process` created successfully. Creating account Toy Account ... Toy account Toy Account with user toy-user@flexmeasures.io created successfully. You might want to run `flexmeasures show account --id 1` Adding transmission zone type ... Adding NL transmission zone ... Created day-ahead prices The sensor recording day-ahead prices is day-ahead prices (ID: 1). Created Created discharging Created Created production The sensor recording battery discharging is discharging (ID: 2). The sensor recording solar forecasts is production (ID: 3). And with that, we're done with the structural data for this tutorial! If you want, you can inspect what you created in the CLI (we'll also show the UI later): .. code-block:: bash $ flexmeasures show account --id 1 =========================== Account Toy Account (ID: 1) =========================== Account has no roles. All users: ID Name Email Last Login Last Seen Roles ---- -------- ------------------------ ------------ ----------- ------------- 1 toy-user toy-user@flexmeasures.io None None account-admin All assets: ID Name Type Parent ID Location ---- ------------ -------- ----------- ----------------- 2 toy-building building 2 (52.374, 4.88969) 3 toy-battery battery 2 (52.374, 4.88969) 4 toy-solar solar 2 (52.374, 4.88969) .. code-block:: bash :emphasize-lines: 9-10 $ flexmeasures show asset --id 2 ========================= Asset toy-building (ID: 2) ========================= Type Location Sensors to show Attributes -------- ----------------- ------------------- ------------ building (52.374, 4.88969) Prices: [1] Power flows: [3, 2] Flex-Context Flex-Model -------------------------------- ------------ site-power-capacity: 500 kVA consumption-price: {'sensor': 1} ==================================== Child assets of toy-building (ID: 2) ==================================== ID Name Type ------- ----------------- ---------------------------- 3 toy-battery battery 4 toy-solar solar No sensors in asset ... You can see that this building asset has some meta information about how FlexMeasures needs to schedule: - Within :ref:`flex_context`, we noted where to find the relevant optimization signal for electricity consumption (Sensor 1, which stores day-ahead prices). - Also, the building has a grid connection capacity of 500 kVA, meaning that the total power flowing into or out of the building cannot exceed this value. Now let's look at the battery asset, as well: .. code-block:: bash :emphasize-lines: 10-12 $ flexmeasures show asset --id 3 =================================== Asset toy-battery (ID: 3) Child of asset toy-building (ID: 2) =================================== Type Location Sensors to show Attributes ------- ----------------- ------------------- ------------ battery (52.374, 4.88969) Prices: [1] Power flows: [3, 2] Flex-Context Flex-Model -------------- ------------------------- power-capacity: 500 kVA roundtrip-efficiency: 90% soc-max: 450 kWh ==================================== Child assets of toy-battery (ID: 3) ==================================== No child assets ... All sensors in asset: ID Name Unit Resolution Timezone Attributes ---- ----------- ------ ------------ ---------------- ------------ 2 discharging MW 15 minutes Europe/Amsterdam Yes, that is quite a large battery :) You can also see that the asset has some meta information about its scheduling. - Within :ref:`flex_model`, we noted that the battery's power capacity is the same as the building's grid connection capacity (500 kVA), meaning the battery can charge or discharge at full power without overloading the connection, but no other devices can (we will come back to this limitation). - Also noted is the battery's roundtrip efficiency (90%) and maximum state of charge (450 kWh). .. note:: Obviously, you can use the ``flexmeasures`` command to create your own, custom account and assets. See :ref:`cli`. And to create, edit or read asset data via the API, see :ref:`v3_0`. We can also look at the battery asset in the UI of FlexMeasures (in Docker, the FlexMeasures web server already runs, on your PC you can start it with ``flexmeasures run``). Visit `http://localhost:5000/ `_ (username is "toy-user@flexmeasures.io", password is "toy-password"): .. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/toy-schedule/asset-view-dashboard.png :align: center | .. note:: You won't see the map tiles, as we have not configured the :ref:`MAPBOX_ACCESS_TOKEN`. If you have one, you can configure it via ``flexmeasures.cfg`` (for Docker, see :ref:`docker_configuration`). And here is the context view of the building: .. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/toy-tutorial-site-structure.png :align: center | The flex-context of the building (which you can edit here in the UI, as well): .. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/toy-tutorial-building-flex-context.png :align: center | And on the flex-model of the battery can be seen on its properties page (and is editable as well): .. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/toy-tutorial-battery-flex-model.png :align: center | .. _tut_toy_schedule_price_data: Add some price data --------------------------------------- Now to add price data. First, we'll create the CSV file with prices (EUR/MWh, see the setup for sensor 1 above) for tomorrow. .. code-block:: bash $ TOMORROW=$(date --date="next day" '+%Y-%m-%d') $ echo "Hour,Price $ ${TOMORROW}T00:00:00,10 $ ${TOMORROW}T01:00:00,11 $ ${TOMORROW}T02:00:00,12 $ ${TOMORROW}T03:00:00,15 $ ${TOMORROW}T04:00:00,18 $ ${TOMORROW}T05:00:00,17 $ ${TOMORROW}T06:00:00,10.5 $ ${TOMORROW}T07:00:00,9 $ ${TOMORROW}T08:00:00,9.5 $ ${TOMORROW}T09:00:00,9 $ ${TOMORROW}T10:00:00,8.5 $ ${TOMORROW}T11:00:00,10 $ ${TOMORROW}T12:00:00,8 $ ${TOMORROW}T13:00:00,5 $ ${TOMORROW}T14:00:00,4 $ ${TOMORROW}T15:00:00,4 $ ${TOMORROW}T16:00:00,5.5 $ ${TOMORROW}T17:00:00,8 $ ${TOMORROW}T18:00:00,12 $ ${TOMORROW}T19:00:00,13 $ ${TOMORROW}T20:00:00,14 $ ${TOMORROW}T21:00:00,12.5 $ ${TOMORROW}T22:00:00,10 $ ${TOMORROW}T23:00:00,7" > prices-tomorrow.csv This is time series data, in FlexMeasures we call *"beliefs"*. Beliefs can also be sent to FlexMeasures via API or imported from open data hubs like `ENTSO-E `_ or `Weather Forecast APIs `_. However, in this tutorial we'll show how you can read data in from a CSV file. Sometimes that's just what you need :) .. code-block:: bash $ flexmeasures add beliefs --sensor 1 --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam Successfully created beliefs In FlexMeasures, all beliefs have a data source. Here, we use the username of the user we created earlier. We could also pass a user ID, or the name of a new data source we want to use for CLI scripts. .. note:: Attention: We created and imported prices where the times have no time zone component! That happens a lot. FlexMeasures can localize them for you to a given timezone. Here, we localized the data to the timezone of the price sensor - ``Europe/Amsterdam`` - so the start time for the first price is `2022-03-03 00:00:00+01:00` (midnight in Amsterdam). Let's look at the price data we just loaded: .. code-block:: bash $ flexmeasures show beliefs --sensor 1 --start ${TOMORROW}T00:00:00+01:00 --duration PT24H Beliefs for Sensor 'day-ahead prices' (ID 1). Data spans a day and starts at 2025-11-11 00:00:00+01:00. The time resolution (x-axis) is an hour. ┌────────────────────────────────────────────────────────────┐ │ ▗▀▚▖ │ │ ▗▘ ▝▖ │ │ ▞ ▌ │ │ ▟ ▐ │ 15EUR/MWh │ ▗▘ ▝▖ ▗ │ │ ▗▘ ▚ ▄▞▘▚▖ │ │ ▞ ▐ ▄▀▘ ▝▄ │ │ ▄▞ ▌ ▛ ▖ │ │▀ ▚ ▐ ▝▖ │ │ ▝▚ ▖ ▗▘ ▝▖ │ 10EUR/MWh │ ▀▄▄▞▀▄▄ ▗▀▝▖ ▞ ▐ │ │ ▀▀▜▘ ▝▚ ▗▘ ▚ │ │ ▌ ▞ ▌│ │ ▝▖ ▞ ▝│ │ ▐ ▞ │ │ ▚ ▗▞ │ 5EUR/MWh │ ▀▚▄▄▄▄▘ │ └────────────────────────────────────────────────────────────┘ 5 10 15 20 ██ day-ahead prices Again, we can also view these prices in the `FlexMeasures UI `_: .. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/toy-schedule/sensor-data-prices.png :align: center | .. note:: Technically, these prices for tomorrow may be forecasts (depending on whether you are running through this tutorial before or after the day-ahead market's gate closure). You can also use FlexMeasures to compute forecasts yourself. See :ref:`tut_forecasting_scheduling`.