push() unit of measure together with signal

[Johannes Sikstrom]

Hi, (updated since first post)

 

I'm trying to push() 2 external signals and metadata for UoM for them in a single push(). SO far i can make it work with 2 separate pushes, one for the data and one for the metadata like in the code below:

 

#create df's
data_for_seeq_FCRD_Up = pd.DataFrame({'FCR-D Up Price': prices_upp}, index=pd.to_datetime(dates_upp))
data_for_seeq_FCRD_Down = pd.DataFrame({'FCR-D Down Price': prices_ned}, index=pd.to_datetime(dates_ned))
# Combine the data for both signals into a single DataFrame
data_for_seeq_combined = pd.concat([data_for_seeq_FCRD_Up, data_for_seeq_FCRD_Down], axis=1)

push_result = spy.push(data=data_for_seeq_combined, workbook='Aitik >> FCR Aitik Monitor', worksheet='FCR Prices')

# Assuming 'push_result' now contains two rows, one for each signal
# Update the 'Name' and 'Value Unit Of Measure' for both signals
push_result.loc[push_result['Name'] == 'FCR-D Up Price', 'Value Unit Of Measure'] = 'EUR/MW'
push_result.loc[push_result['Name'] == 'FCR-D Down Price', 'Value Unit Of Measure'] = 'EUR/MW'

# Push the combined metadata back to Seeq
push_results_meta_combined = spy.push(metadata=push_result, workbook='Aitik >> FCR Aitik Monitor', worksheet='FCR Prices')

# Print the combined push results
print(push_results_meta_combined)

When i try to structure the metadata and send in in the same push() as the data it looks like this:
 

data_for_seeq_FCRD_Up = pd.DataFrame({'FCR-D Up Price': prices_upp}, index=pd.to_datetime(dates_upp))
data_for_seeq_FCRD_Down = pd.DataFrame({'FCR-D Down Price': prices_ned}, index=pd.to_datetime(dates_ned))
# Combine the data for both signals into a single DataFrame
data_for_seeq_combined = pd.concat([data_for_seeq_FCRD_Up, data_for_seeq_FCRD_Down], axis=1)

# Create a single metadata DataFrame for both signals
metadata_combined = pd.DataFrame({
    'Name': ['FCR-D Up Price', 'FCR-D Down Price'],  # Signal names
    'Value Unit Of Measure': ['EUR/MW', 'EUR/MW']  # Units of measure for both signals
    # Add other required fields for metadata as per your requirements
})
metadata_combined.index = metadata_combined['Name']
print(data_for_seeq_combined.head())
print(metadata_combined.head())

push_results_combined = spy.push(data=data_for_seeq_combined, metadata=metadata_combined, workbook='Aitik >> FCR Aitik Monitor', worksheet='FCR Prices',datasource='Mimer SVK API')

The format of the data and metadata dataframes looks like:
 

                           FCR-D Up Price  FCR-D Down Price
2023-11-16 00:00:00+01:00       23.066646          9.116699
2023-11-16 01:00:00+01:00       22.670452          9.076252
2023-11-16 02:00:00+01:00       21.816880          8.906874
2023-11-16 03:00:00+01:00       21.654117          8.916882
2023-11-16 04:00:00+01:00       26.594917          8.885253
                              Name Value Unit Of Measure
Name                                                    
FCR-D Up Price      FCR-D Up Price                EUR/MW
FCR-D Down Price  FCR-D Down Price                EUR/MW

 

The error I keep getting is:

SPy Error: Items with no valid type specified cannot be pushed unless they are calculations. "Formula" column is required for such items.

 

So when I send then in separate pushes it works, but sending data and metadata together then the type is not valid for the signals? What is happening here?

Edited November 17, 2023 by Johannes Sikstrom

[Synjen Marrocco]

Hi Johannas,

It looks like in your first post that your metadata dataframe might have an issue. Can you try the same code but remove the inplace= True from the set_index? Additionally, can you add a 'Type' column to the metadata? You can also output the metadata frame before the push to ensure it is as expected.

This example seemed to work for me:

from seeq import spy
import pandas as pd

import csv
csv_file = pd.read_csv('data_.csv', parse_dates=['Date-Time'], index_col='Date-Time')
    
metadata = pd.DataFrame({
    'Name': ['Temperature', 'RH'], 
    'Value Unit Of Measure': 'EUR/MW',
    'Type':'Signal'

}).set_index('Name', drop=False)

spy.push(data = csv_file, metadata = metadata, workbook='synjen_test004')

 

 

[Johannes Sikstrom]

Hi,

yes! Those changes seem to do the trick. I now have this that works:

data_for_seeq_FCRD_Up = pd.DataFrame({'FCR-D Up Price': prices_upp}, index=pd.to_datetime(dates_upp))
data_for_seeq_FCRD_Down = pd.DataFrame({'FCR-D Down Price': prices_ned}, index=pd.to_datetime(dates_ned))
# Combine the data for both signals into a single DataFrame
data_for_seeq_combined = pd.concat([data_for_seeq_FCRD_Up, data_for_seeq_FCRD_Down], axis=1)

# Create a single metadata DataFrame for both signals
metadata_combined = pd.DataFrame({
    'Name': ['FCR-D Up Price', 'FCR-D Down Price'],  # Signal names
    'Value Unit Of Measure': 'EUR/MW',  # Units of measure for both signals
    'Type':'Signal'
    # Add other required fields for metadata as per your requirements
}).set_index('Name', drop=False)

# Push the combined data and metadata to Seeq in one operation
push_results_combined = spy.push(data=data_for_seeq_combined, metadata=metadata_combined, workbook='Aitik >> FCR Aitik Monitor', worksheet='FCR Prices',datasource='Mimer SVK API')

I thought I had tried all combinations but I suspect the 'Type': 'Signal' was the culprit. In my attempts to add Type I added just the short form 'Si' (for 'Signal') and that does not work. That's why I removed the Type assign from the dataframe.

Thank you!