Bsrgvty

I encountered some statisticians that never use models other than Linear Regression for prediction because they believe that "ML models" such as random forest or gradient boosting are hard to explain or "not interpretable".

In a Linear Regression, given that the set of assumptions is verified (normality of errors, homoskedasticity, no multi-collinearity), the t-tests provide a way to test the significance of variables, tests that to my knowledge are not available in the random forests or gradient boosting models.

Therefore, my question is if I want to model a dependent variable with a set of independent variables, for the sake of interpretability should I always use Linear Regression?

It is hard for me to believe that you heard people saying this, because it would be a dumb thing to say. It's like saying that you use only the hammer (including drilling holes and for changing the lightbulbs), because it's straightforward to use and gives predictable results.

Second, linear regression is not always "interpretable". If you have linear regression model with many polynomial terms, or just a lot of features, it would be hard to interpret. For example, say that you used the raw values of each of the 784 pixels from MNIST† as features. Would knowing that pixel 237 has weight equal to -2311.67 tell you anything about the model? For image data, looking at activation maps of the convolutional neural network would be much easier to understand.

Finally, there are models that are equally interpretable, e.g. logistic regression, decision trees, naive Bayes algorithm, and many more.

† - As noticed by Ingolifs in the comment, and as discussed in this thread, MNIST may be not the best example, since this is a very simple dataset. For most of the realistic image datasets, logistic regression would not work and looking at the weights would not give any straightforward answers. If you look closer at the weights in the linked thread, then their interpretation is also not straightforward, for example weights for predicting "5" or "9" do not show any obvious pattern (see image below, copied from the other thread).

Decision Tree would be another choice. Or Lasso Regression to create a sparse system.

Check this figure from An Introduction to Statistical Learning book.

http://www.sr-sv.com/wp-content/uploads/2015/09/STAT01.png

No, that is needlessly restrictive. There are a large range of interpretable models including not just (as Frans Rodenburg says) linear models, generalized linear models and generalized additive models, but also machine learning methods used for regression. I include random forests, gradient boosted machines, neural networks, and more. Just because you don't get coefficients out of machine learning models that are similar to those from linear regressions does not mean that their workings cannot be understood. It just takes a bit more work.

To understand why, I'd recommend reading this question: Obtaining knowledge from a random forest . What it shows is how you can approach making almost any machine learning model interpretable.

I would agrre with Tim's and mkt's answers - ML models are not necessarily uninterpretable. I would direct you to the Descriptive mAchine Learning EXplanations, DALEX R package, which is devoted to making ML models interpretable.