{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "_quarto:\n",
    "  tests:\n",
    "    html:\n",
    "      ensureHtmlElements:\n",
    "        - ['.quarto-title h1.title']    \n",
    "        - ['main section > h1']\n",
    "---"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "ZnZV4XfHSPcw"
   },
   "source": [
    "# Quickstart for Customer Churn Model Documentation — Full Suite\n",
    "\n",
    "This interactive notebook guides model developers through the process of documenting a model with the ValidMind Developer Framework. It uses the [Bank Customer Churn Prediction](https://www.kaggle.com/code/kmalit/bank-customer-churn-prediction/data) sample dataset from Kaggle to train a simple classification model.\n",
    "\n",
    "As part of the notebook, you will learn how to train a sample model while exploring how the documentation process works:\n",
    "\n",
    "- Initializing the ValidMind Developer Framework\n",
    "- Loading a sample dataset provided by the library to train a simple classification model\n",
    "- Running a ValidMind test suite to quickly generate documention about the data and model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ValidMind at a glance\n",
    "\n",
    "ValidMind's platform enables organizations to identify, document, and manage model risks for all types of models, including AI/ML models, LLMs, and statistical models. As a model developer, you use the ValidMind Developer Framework to automate documentation and validation tests, and then use the ValidMind AI Risk Platform UI to collaborate on documentation projects. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.\n",
    "\n",
    "If this is your first time trying out ValidMind, you can make use of the following resources alongside this notebook:\n",
    "\n",
    "- [Get started](https://docs.validmind.ai/guide/get-started.html) — The basics, including key concepts, and how our products work\n",
    "- [Get started with the ValidMind Developer Framework](https://docs.validmind.ai/guide/get-started-developer-framework.html) —  The path for developers, more code samples, and our developer reference"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "ZNcbDRubSPc1"
   },
   "source": [
    "## Before you begin\n",
    "\n",
    "::: {.callout-tip}\n",
    "### New to ValidMind? \n",
    "For access to all features available in this notebook, create a free ValidMind account. \n",
    "\n",
    "Signing up is FREE — [**Sign up now**](https://app.prod.validmind.ai)\n",
    ":::\n",
    "\n",
    "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install the client library\n",
    "\n",
    "The client library provides Python support for the ValidMind Developer Framework. To install it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6G5-kHOZ7YWk"
   },
   "outputs": [],
   "source": [
    "%pip install -q validmind"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "_ZqI8W5jSPc1"
   },
   "source": [
    "## Initialize the client library\n",
    "\n",
    "Every documentation project in the Platform UI comes with a _code snippet_ that lets the client library associate your documentation and tests with the right project on the Platform UI when you run this notebook. As you will see later, documentation projects are useful because they act as containers for model documentation and validation reports and they enable you to organize all of your documentation work in one place. \n",
    "\n",
    "Get your code snippet by creating a documentation project:\n",
    "\n",
    "1. In a browser, log into the [Platform UI](https://app.prod.validmind.ai).\n",
    "\n",
    "2. Go to **Documentation Projects** and click **Create new project**.\n",
    "\n",
    "3. Select **`[Demo] Customer Churn Model`** and **`Initial Validation`** for the model name and type, give the project a unique  name to make it yours, and then click **Create project**.\n",
    "\n",
    "4. Go to **Documentation Projects** > **YOUR_UNIQUE_PROJECT_NAME** > **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, replace this placeholder with your own code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5hqGn9jHSPc2"
   },
   "outputs": [],
   "source": [
    "## Replace this placeholder with the code snippet from your own project ##\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "  api_host = \"https://api.prod.validmind.ai/api/v1/tracking\",\n",
    "  api_key = \"...\",\n",
    "  api_secret = \"...\",\n",
    "  project = \"...\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize the Python environment\n",
    "\n",
    "Next, let's import the necessary libraries and set up your Python environment for data analysis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost as xgb\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preview the documentation template\n",
    "\n",
    "A template predefines sections for your documentation project and provides a general outline to follow, making the documentation process much easier.\n",
    "\n",
    "You will upload documentation and test results into this template later on. For now, take a look at the structure that the template provides with the `vm.preview_template()` function from the ValidMind library and note the empty sections:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.preview_template()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "WT4iDaNPSPc4"
   },
   "source": [
    "## Load the sample dataset\n",
    "\n",
    "The sample dataset used here is provided by the ValidMind library. To be able to use it, you need to import the dataset and load it into a pandas [DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html), a two-dimensional tabular data structure that makes use of rows and columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BlNanClPSPc5"
   },
   "outputs": [],
   "source": [
    "# Import the sample dataset from the library\n",
    "\n",
    "from validmind.datasets.classification import customer_churn as demo_dataset\n",
    "\n",
    "print(f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{demo_dataset.target_column}' \\n\\t• Class labels: {demo_dataset.class_labels}\")\n",
    "\n",
    "raw_df = demo_dataset.load_data()\n",
    "raw_df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "sP6OZpdGSPc6"
   },
   "source": [
    "## Document the model\n",
    "\n",
    "As part of documenting the model with the ValidMind Developer Framework, you need to preprocess the raw dataset, initialize some training and test datasets, initialize a model object you can use for testing, and then run the full suite of tests. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "XNI6mCy0SPc6"
   },
   "source": [
    "### Prepocess the raw dataset\n",
    "\n",
    "Preprocessing performs a number of operations to get ready for the subsequent steps:\n",
    "\n",
    "- Preprocess the data: Splits the DataFrame (`df`) into multiple datasets (`train_df`, `validation_df`, and `test_df`) using `demo_dataset.preprocess` to simplify preprocessing.\n",
    "- Separate features and targets: Drops the target column to create feature sets (`x_train`, `x_val`) and target sets (`y_train`, `y_val`).\n",
    "- Initialize XGBoost classifier: Creates an `XGBClassifier` object with early stopping rounds set to 10.\n",
    "- Set evaluation metrics: Specifies metrics for model evaluation as \"error,\" \"logloss,\" and \"auc.\"\n",
    "- Fit the model: Trains the model on `x_train` and `y_train` using the validation set `(x_val, y_val)`. Verbose output is disabled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PMeDVcpsSPc7"
   },
   "outputs": [],
   "source": [
    "train_df, validation_df, test_df = demo_dataset.preprocess(raw_df)\n",
    "\n",
    "x_train = train_df.drop(demo_dataset.target_column, axis=1)\n",
    "y_train = train_df[demo_dataset.target_column]\n",
    "x_val = validation_df.drop(demo_dataset.target_column, axis=1)\n",
    "y_val = validation_df[demo_dataset.target_column]\n",
    "\n",
    "model = xgb.XGBClassifier(early_stopping_rounds=10)\n",
    "model.set_params(\n",
    "    eval_metric=[\"error\", \"logloss\", \"auc\"],\n",
    ")\n",
    "model.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    eval_set=[(x_val, y_val)],\n",
    "    verbose=False,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "DTO0bN4qSPc7"
   },
   "source": [
    "### Initialize the ValidMind datasets\n",
    "\n",
    "Before you can run tests, you must first initialize a ValidMind dataset object using the [`init_dataset`](https://docs.validmind.ai/validmind/validmind.html#init_dataset) function from the ValidMind (`vm`) module. \n",
    "\n",
    "This function takes a number of arguments:\n",
    "\n",
    "- `dataset` — the raw dataset that you want to provide as input to tests\n",
    "- `input_id` - a unique identifier that allows tracking what inputs are used when running each individual test\n",
    "- `target_column` — a required argument if tests require access to true values. This is the name of the target column in the dataset\n",
    "- `class_labels` — an optional value to map predicted classes to class labels\n",
    "\n",
    "With all datasets ready, you can now initialize the raw, training and test datasets (`raw_df`, `train_df` and `test_df`) created earlier into their own dataset objects using [`vm.init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ShiOFS7bSPc7"
   },
   "outputs": [],
   "source": [
    "vm_raw_dataset = vm.init_dataset(\n",
    "    dataset=raw_df,\n",
    "    input_id='raw_dataset',\n",
    "    target_column=demo_dataset.target_column,\n",
    "    class_labels=demo_dataset.class_labels\n",
    ")\n",
    "\n",
    "vm_train_ds = vm.init_dataset(\n",
    "    dataset=train_df,\n",
    "    input_id='train_dataset',\n",
    "    target_column=demo_dataset.target_column\n",
    ")\n",
    "\n",
    "vm_test_ds = vm.init_dataset(\n",
    "    dataset=test_df,\n",
    "    input_id='test_dataset',\n",
    "    target_column=demo_dataset.target_column\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "l6dpArGCSPc7"
   },
   "source": [
    "### Initialize a model object \n",
    "\n",
    "Additionally, you need to initialize a ValidMind model object (`vm_model`) that can be passed to other functions for analysis and tests on the data. You simply intialize this model object with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wE0OckXjSPc7"
   },
   "outputs": [],
   "source": [
    "vm_model = vm.init_model(\n",
    "    model,\n",
    "    input_id=\"model\",\n",
    "    train_ds=vm_train_ds,\n",
    "    test_ds=vm_test_ds,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "8xaBL0_3SPc7"
   },
   "source": [
    "### Run the full suite of tests\n",
    "\n",
    "This is where it all comes together: you are now ready to run the documentation tests for the model as defined by the documentation template you looked at earlier. \n",
    "\n",
    "The [`vm.run_documentation_tests`](https://docs.validmind.ai/validmind/validmind.html#run_documentation_tests) function finds and runs every test specified in the template and then uploads all the documentation and test artifacts that get generated to the ValidMind AI Risk Platform. \n",
    "\n",
    "The function takes two arguments:\n",
    "\n",
    "- `dataset`: The data to be tested, specified as `vm_dataset`.\n",
    "- `model`: The model to be used for testing, specified as `vm_model`.\n",
    "\n",
    "The variable `full_suite` then holds the result of these tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NgzKVN_gSPc8"
   },
   "outputs": [],
   "source": [
    "full_suite = vm.run_documentation_tests(\n",
    "    inputs = {\n",
    "        \"dataset\": vm_raw_dataset,\n",
    "        \"model\": vm_model\n",
    "    }\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the test results as part of your model documentation right in the ValidMind Platform UI: \n",
    "\n",
    "1. Log back into the [Platform UI](https://app.prod.validmind.ai) \n",
    "\n",
    "2. Go to **Documentation Projects** > **YOUR_UNIQUE_PROJECT_NAME** > **Documentation**.\n",
    "\n",
    "3. Expand the following sections and take a look around:\n",
    "   \n",
    "   - **2. Data Preparation**\n",
    "   - **3. Model Development**\n",
    "\n",
    "What you can see now is a much more easily consumable version of the documentation, including the results of the tests you just performed, along with other parts of your documentation project that still need to be completed. There is a wealth of information that gets uploaded when you run the full test suite, so take a closer look around, especially at test results that might need attention (hint: some of the tests in 2.1 Data description look like they need some attention).\n",
    "\n",
    "If you want to learn more about where you are in the model documentation process, take a look at [How do I use the framework?](https://docs.validmind.ai/guide/get-started-developer-framework.html#how-do-i-use-the-framework).\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
