Import/Process/Write data with SCILAB

SCILAB is a powerful tool enabling you to develop technically advanced applications in domain such signal processing, optimization, control system and so on. That's pretty cool. But tell me ... why is it so hard to import text data? Actually, because one can put anything in any format within an ASCII file. So, what should you do, and what functions should you use if you want to import text data in SCILAB? Here are some tips and good practices I got practising on the subject of experimental/simulation data import.

A zip file gathering all the tutorial files is provided above for download.

Data Import

The most basic function: fscanfMat

In the ideal case of your data file is well formated (no unnecessary blank space) and gathering only data stored as matrix (could have a header), what you can do is leverage the fscanfMat function to import your text file in a SCILAB matrix.

But as you will experience it, this function is not suitable for every text file.

A more flexible function: csvRead

When you are looking for more flexibility (selecting range for rows and columns, conversion into double/string, ...), or you want to access CSV data (coming from an excel file for example) ... or when fscanfMat is not working, you can use the csvRead function.

In this case, we are reading the three text files provided in the zip archive.

• For the first one which mainly about characters, we impose a conversion to string waiting for the full file to be further processed within SCILAB.

• For the second one we already saw, we specify tghe number of lines at the beginning of the file to be ignored: 1 corresponding to the header. This allows us to make direct conversion of data into double (default).
• For the third one which is a CSV file, we specify separator (";") and decimal (",") as well as range of data to be imported: from line 2 to 5, from column 2 to 4.

When nothing else is working: mopen/mgetl

Sometimes it is just not enough. Sometimes, text files are following the physical law of thermodynamics (especially the second one). In those critical case there is one last solution.

Use mopen function to open the text file (see help for different options), use mgetl function to get all the lines as array of strings and DO NOT FORGOT TO CLOSE YOUR FILE using mclose function. Whenever your script run into an error and do not reach the mclose command line, you will have to close your file manually (>> mclose(id) in the console) otherwise you won't be able to access your file until SCILAB releases it.

In the case of Data1.txt file, this function is providing the same output as the csvRead function.

Text data processing

Now let's say we are in the worst case scenario. You did manage to import your data within SCILAB (did you?) however you got the full text as an array of string instead of the needed data (what could be provided by csvRead function with string conversion or mopen/mgetl). If we take the example of Data1.txt file (see Var - A image above). This file is mainly about text information but is also providing a few quantitative information regarding some sensors. Let’s see how to get sensors IDs (2341 and 453) and coordinates ([3 2 1] and [-22 3 1]).

First of all, import text within a SCILAB matrix, using default csvRead function, settings for separators (" ") and decimals (".") and forcing conversion to string (see image above).

Locate substring: strstr & find

Based on the results, one can locate the %SENSOR sub-string. To do so you can use the strstr function which will provide you with an array of string, with empty lines where there is no match with searched pattern.

And finally get index of interesting lines (non empty one) using the find function.

Split text into parts: strsplit & strsubst

Next step is to split the two lines looking for blank space using strsplit function. ID are in second position, POSITIONS are in 4th, 5th and 6th position.

At this stage, IDs is an array of string but containing only numbers. But POSs is an array of string containing numbers and parenthesis. To remove those parenthesis, you can use the strsubst function, specifying you want to substitute parenthesis pattern with empty string.

String conversion into double: evstr

Final step is about converting the string representing numbers into doubles for further processing. This can easily be done using the evstr function.

As a results, we got to matrix of doubles

Export Data

Save SCILAB data: save/load

Now that you have imported and processed your data in SCILAB, you would fine handy to save those instead of doing this tedious process any time you want to access your data. Best is to save SCILAB variables as SCILAB specific binary data format SOD using save function.

Once this is done, you can access your data anaytime using the load function and specifying the name of the file you want to import. Note that the variable name might be different from the file name.

Write text file: csvWrite

In case you want to write a text file this time, best is to use the csvWrite function. You first will have to concatenate your data as an array of strings.