Configuration File for Knowledge-Debugger

The default Knowledge-Debugger configuration-file looks like this:

server_files_dir: "lm-debugger/server_files/",
model_name: "hf_organization/model_name",
device: "cuda:0",
server_ip: "127.0.0.1",
server_port: 8000,
elastic_ip: "127.0.0.1",
elastic_port: 9200,
react_ip: "127.0.0.1",
react_port: 3000,
streamlit_ip: "127.0.0.1",
streamlit_port: 8501,
top_k_tokens_for_ui: 10,
top_k_for_elastic: 50,
num_layers: 32,
elastic_index: "model_projections_docs",
elastic_projections_path: $.server_files_dir + "my_file.pkl",
elastic_api_key: "VGhlIGNha2UgaXMgYSBsaWU=",
enable_caching_weight_matrix: true,

layer_mappings: {
    base_model_descriptor: "transformer",
    token_embedding: "transformer.wte",
    mlp_sublayer: "transformer.h.{}.mlp",
    attn_sublayer: "transformer.h.{}.attn",
    mlp_activations: "transformer.h.{}.mlp.act",
    mlp_gate_proj: "transformer.h.{}.mlp.c_fc",
    mlp_up_proj: "transformer.h.{}.mlp.c_fc",
    mlp_down_proj: "transformer.h.{}.mlp.c_proj",
    decoder_input_layernorm: "transformer.h.{}.ln_1",
    decoder_post_attention_layernorm: "transformer.h.{}.ln_2",
    post_decoder_norm: "transformer.ln_f"
  },

  easy_edit_hparams_path: "/nfs/home/ebenz_bsc2024/lm-debugger/config_files/ee_hparams/gpt2-xl",

  metric_configs: {
    NormOfWeightUpdate: {},
    CausalTrace: {
      # Config for Causal Trace
      samples: 10,
      noise: 0.1,
      window: 10,
      kind: "mlp"
    },
    DenominatorOfROMEUpdate: {
      applicable_intervention_methods: ["ROME", "R-ROME"]
    },
    Perplexity: {
      # From https://github.com/WanliYoung/Collapse-in-Model-Editing/tree/main
      dataset: []
    }
  }

Explanation of the datafields:

• server_files_dir: Directory for the Server Files to be stored. This includes the forward Projections of the MLP (LMDebuggerIntervention)

• model_name: HuggingFace-Name of the Model. The Model-Name must be supported by the used Instance of TransformerModelWrapper

• device: Device, the Transformer is loaded to (e.g., “cuda”, “cuda:0”, “cpu”)

• server_ip: Flask-Backend Server IP

• server_port: Flask-Backend Server Port

• elastic_ip: ElasticSearch Server IP

• elastic_port: ElasticSearch Server Port

• react_ip: React-Frontend IP

• react_port: React-Frontend Port

• streamlit_ip: Streamlit-Frontend IP

• streamlit_port: Streamlit-Frontend Port

• top_k_tokens_for_ui: Number of Tokens that are shown in the React-Frontend (as most probable Tokens before and after Interventions)

• top_k_for_elastic: Number of Tokens, returned by each ElasticSearch-Request

• num_layers: Number of Layers of Transformer

• elastic_index: Name of ElasticSearch-Index, the MLP-Projections are stored in

• elastic_projections_path: Path to the Pickle-File containing the MLP-Projections

  • lm_debugger/es_client/create_offline_files.py: Generates this Pickle-File

  • lm_debugger/es_client/index_value_projections_docs.py: Uploads the Pickle-File to the ElasticSearch-Instance

• elastic_api_key: API-Key of the ElasticSearch-Instance

• enable_caching_weight_matrix: Enables Caching of the Weight-Updates done by Sequences of Interventions. Saves compute on repetitive Runs on the same Sequence of Interventions.

• layer_mappings: Mappings of Layers to their Descriptors. Used in Interventions. (“{}” refers to Layer-Index)

• easy_edit_hparams_path: Path to directory of Hparams-yaml-Files of each EasyEdit-Interventions Method, used in this app

• metric_configs: Configurations of each Metric. The Config-Dicts for each Metric are accessible in the Metric via the config-Attribute