Generate design matrix

The model we are trying to fit is in the following form volatility = β₀ volume ^β₁. Taking the log on both sides we get: log(volatility) = log(β₀) + β₁ log(volume). We rename log(β₀) = β₀’ since this is a fixed effect for each of the stocks to control for the baseline effect. The final formula is the following:

log(volatility) = β₀' + β₁ log(volume)

β₁ is a scalar and represents the slope of the regression line that we are going to fit. For β₁>1, we will find that the function volatility = β₀ volume ^β₁ is convex, which means that volatility grows much faster as volume increases.

Apply `OneHotEncoder` to Categorical Columns

Spark and Pyspark provide several feature transformation and extraction functions. They can be recovered in the library pyspark.ml.feature. The following document Spark feature transformers has the complete list and description for Spark 2.2.0 with both examples in Scala and Python. In the following case, we are trying to encode the column Mnemonic–which is a string containing 3 or 4-letter abbreviation of the stock name, into dummy variables with the following structure:

Mnemonic	SANT	1FC	OMV
SANT	1.0	0.0	0.0
SANT	1.0	0.0	0.0
1FC	0.0	1.0	0.0
OMV	0.0	0.0	1.0

Pyspark achieves this result by first using StringIndexer, transforming strings into numerical indices, and then tranforming indices into a sparse vector with OneHotEncoder as shown in the following paragraph in the column categoryVec. The sparse vector is a way to compress the data if compared to the table above. In the table above, most of the matrix of dummy variables is composed of 0’s, which is lend themselves to compression. The sparse vector has the following format:

([length of the vector], [column index for non-zero elements of the vector], [value for the non-zero elements of the vector])

This will download a JSON file of the Notebook that can be imported into your Qubole account to test yourself.

Don't have a Qubole account? Click here for a free Qubole Test Drive