`SklearnTransformerWrapper` fails when FunctionTransformer used with non-numerical columns

[david-cortes]

The SklearnTransformerWrapper will throw an error if I try to use it for non-numerical columns, even though some scikit-learn transformers are able to handle non-numerical columns.

Consider the following example:

import numpy as np, pandas as pd
from sklearn.preprocessing import FunctionTransformer
from feature_engine.wrappers import SklearnTransformerWrapper

df = pd.DataFrame({
    "col1" : ["1", "2", "3"],
    "col2" : ["a", "b", "c"]
})

pipeline = SklearnTransformerWrapper(
    FunctionTransformer(lambda x: x.astype(np.float64)),
    variables=["col1"]
)
pipeline.fit(df)

TypeError                                 Traceback (most recent call last)
<ipython-input-1-88807bab80e9> in <module>
     12     variables=["col1"]
     13 )
---> 14 pipeline.fit(df)

~/anaconda3/envs/py3/lib/python3.9/site-packages/feature_engine/wrappers/wrappers.py in fit(self, X, y)
    122 
    123         else:
--> 124             self.variables_ = _find_or_check_numerical_variables(X, self.variables)
    125 
    126         self.transformer_.fit(X[self.variables_], y)

~/anaconda3/envs/py3/lib/python3.9/site-packages/feature_engine/variable_manipulation.py in _find_or_check_numerical_variables(X, variables)
     75         # check that user entered variables are of type numerical
     76         if any(X[variables].select_dtypes(exclude="number").columns):
---> 77             raise TypeError(
     78                 "Some of the variables are not numerical. Please cast them as "
     79                 "numerical before using this transformer"

TypeError: Some of the variables are not numerical. Please cast them as numerical before using this transformer

[solegalli]

Hi @david-cortes

Thank you for bringing this up. It was out of my radar.

It should be an easy fix though, we would just need to add the FunctionTransformer (and any other class that you are aware that can take categorical variables) to the list that allows that behaviour within the SklearnTransformerWrapper.

Would you like to give it a go?

[david-cortes]

I'm not sure that having a list of exceptions is the right way to approach the problem. One might want to use SklearnTransformerWrapper with user-defined transformers that are scikit-learn compatible too.

[solegalli]

What would you suggest?

[david-cortes]

(sorry, seems somehow my earlier comment got lost)

I think it’d be better to avoid this check altogether and let the transformer throw an error instead. I guess most scikit-learn transformers will be able to throw an error in such cases, even if the error message is not as informative.

[solegalli]

While I agree that having no check for variable types would be easier to implement and to maintain, it goes a bit against Feature-engine's spirit, where we try to enforce applying suitable transformations to adequate variable types.

[david-cortes]

Perhaps then it could have a boolean switch for whether to perform this check or not.

[solegalli]

I think, the easiest solution for this issue would be to add the FunctionTransformer to the list of transformers that can take both categorical and numerical variables as input.

From a code perspective, this would be a very minor modification and would solve the issue raised here.

If more of these type of issues arise, we can think of introducing the switch you suggest.

[solegalli]

The needed change includes adding the string "FunctionTransformer" in this list and then write a small test at the very end of this script with the following test:

X = pd.DataFrame({
    "col1" : ["1", "2", "3"],
    "col2" : ["a", "b", "c"]
})

X_expected = pd.DataFrame({
    "col1" : [1.0, 2.0, 3.0],
    "col2" : ["a", "b", "c"]
})
transformer = SklearnTransformerWrapper(
    FunctionTransformer(lambda x: x.astype(np.float64)),
    variables=["col1"]
)
X_tf = transformer.fit_transform(X)

pd.assert_frame_equal(X_expected, X_tr)

I add in the amount of detail as this is a suitable issue for a first timer :)

[nandevers]

@solegalli

I am a newcomer and I'd be thrilled to contribute to the project.

I wrote the test function with a few tweaks to the above content:

def test_sklearn_transformer_wrapper():
    X = pd.DataFrame({
        "col1": ["1", "2", "3"],
        "col2": ["a", "b", "c"]
        })

    X_expected = pd.DataFrame({
        "col1": [1.0, 2.0, 3.0],
        "col2": ["a", "b", "c"]
        })

    transformer = SklearnTransformerWrapper(
        FunctionTransformer(lambda x: x.astype(np.float64)),
        variables=["col1"]
    )

    X_tf = transformer.fit_transform(X)

    pd.testing.assert_frame_equal(X_expected, X_tf)

I'll be adding a PR shortly, would you please be so kind as to help me with the review?

[david-cortes]

But even if it is made to work for FunctionTransformer, it still wouldn't work for user-defined transformers. I find myself very often defining custom transformers that I want to use with SklearnTransformerWrapper such as this:

from sklearn.base import BaseEstimator, TransformerMixin

class AddMissingToCateg(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self
    def transform(self, X):
        cols_categ = X.columns[X.dtypes == "category"]
        X = X.copy()
        for cl in cols_categ:
            if "unknown" not in X[cl].cat.categories:
                X[cl] = X[cl].cat.add_categories(["unknown"])
            mask_na = pd.isnull(X[cl])
            X[cl].loc[mask_na] = "unknown"
        return X

This transformer is not part of scikit-learn but it follows its idioms and is compatible with pipelines. Yet, it would still fail.