added `wipe` make command (#1215)

* added `wipe` make command

* Apply suggestions from code review

Thanks for your suggestions, I like to apply them.

Co-authored-by: lopagela <lpglm@orange.fr>

* added `wipe` command to the documentation

* rebased to generate valid openapi.json

---------

Co-authored-by: lopagela <lpglm@orange.fr>

Makefile

@@ -49,4 +49,7 @@ api-docs:
 	poetry run python scripts/extract_openapi.py private_gpt.main:app --out docs/openapi.json
 
 ingest:
-	@poetry run python scripts/ingest_folder.py $(call args)
+	@poetry run python scripts/ingest_folder.py $(call args)
+
+wipe:
+	poetry run python scripts/utils.py wipe

README.md

@@ -136,7 +136,6 @@ you want to give a hand:
 - Better observability of the RAG pipeline
 
 ### Project Infrastructure
-- Create a “wipe” shortcut in `make` to remove all contents of local_data folder except .gitignore
 - Packaged version as a local desktop app (windows executable, mac app, linux app)
 - Dockerize the application for platforms outside linux (Docker Desktop for Mac and Windows)
 - Document how to deploy to AWS, GCP and Azure.

docs/description.md

@@ -462,6 +462,11 @@ or using the completions / chat API.
 When running in a local setup, you can remove all ingested documents by simply
 deleting all contents of `local_data` folder (except .gitignore).
 
+To simplify this process, you can use the command:
+```bash
+make wipe
+```
+
 ## API
 
 As explained in the introduction, the API contains high level APIs (ingestion and chat/completions) and low level APIs

scripts/utils.py

@@ -0,0 +1,37 @@
+import argparse
+import os
+import shutil
+
+
+def wipe():
+    path = "local_data"
+    print(f"Wiping {path}...")
+    all_files = os.listdir(path)
+
+    files_to_remove = [file for file in all_files if file != ".gitignore"]
+    for file_name in files_to_remove:
+        file_path = os.path.join(path, file_name)
+        try:
+            if os.path.isfile(file_path):
+                os.remove(file_path)
+            elif os.path.isdir(file_path):
+                shutil.rmtree(file_path)
+            print(f" - Deleted {file_path}")
+        except PermissionError:
+            print(
+                f"PermissionError: Unable to remove {file_path}. It is in use by another process."
+            )
+            continue
+
+
+if __name__ == "__main__":
+    commands = {
+        "wipe": wipe,
+    }
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "mode", help="select a mode to run", choices=list(commands.keys())
+    )
+    args = parser.parse_args()
+    commands[args.mode.lower()]()