Release v0.6

Add shortcuts to make entering queries more efficient

CTRL+F: Highlights rightmost filter, e. g. c:(word1 word2) would
highlight everything between (), so "word1 word" here. Alternatively,
highlights lone words, so p:(docs) word1, would highlight word1.

CTRL+W: Removes last filter or words.

CHANGELOG.md

@@ -1,7 +1,22 @@
 # looqs: Release notes
+## 2022-08-14 - v0.6
+This release features multiple fixes and enhancements.
+
+Bad news first: It drops a trivial trigger that appeared to work quite fine, but silently may cause "unpredictability" of the sqlite FTS5 index ( [9422a5b494](https://github.com/quitesimpleorg/looqs/commit/9422a5b494dabd0f1324dc2f92a34c3036137414) ). As a result, FTS queries may return weird and unexplainable results. This is not reasonably automatically recoverable by looqs. I strongly recommend creating a clean, new database. All previous versions are affected. To do that, go to "Settings" -> checking "Remove old database on save" -> "Save settings and restart". Alternatively, specify a new path to keep the old database.
+
+CHANGES:
+
+ - GUI: Add line numbers and context lines to plaintext previews
+ - GUI: Fix case where previews for old queries would have still been visible if new query would not create previews
+ - GUI: Add CTRL + F, CTRL+W, CTRL+Tab, CTRL+Shift+Tab shortcuts (see user manual)
+ - GUI: Add checkbox in "Settings" tab allowing to delete database.
+ - General: Fix wrong regexes that caused query errors with chars like -
+ - General: Drop trigger sending incomplete sqlite fts5 deletion command, causing undefined index behaviour
+
 ## 2022-07-30 - v0.5.1
 
 CHANGES:
+
  - gui: Fix regression in implicit paths queries introduced in previous version
 
 ## 2022-07-29 - v0.5

README.md

@@ -1,10 +1,9 @@
 # looqs - Full-text search with previews for your files
-looqs is a tool that creates a full-text search index for your files. It allows you to look at previews where your
+looqs is a tool that creates a full-text search index for your files. It allows you to look at previews where your search terms have been found, as shown in the screenshots below.
-search terms have been found, as shown in the screenshots below.
 
 ## Screenshots
 ### Preview
-looqs allow you to look inside files. It marks what you have searched for.
+looqs allows you to look inside files. It highlights what you have searched for.
 
 ![Screenshot looqs](https://garage.quitesimple.org/assets/looqs/orwell.png)
 ![Screenshot looqs search fstream](https://garage.quitesimple.org/assets/looqs/fstream_write.png)
@@ -29,7 +28,7 @@ There is no need to write the long form of filters. There are also booleans avai
 The screenshots in this section may occasionally be slightly outdated, but they are usually recent enough to get an overall impression of the current state of the GUI.
 
 ## Current status
-Latest version: 2022-07-30, v0.5.1
+Latest version: 2022-08-14, v0.6
 
 Please see [Changelog](CHANGELOG.md) for a human readable list of changes.
 

USAGE.md

@@ -9,6 +9,8 @@ looqs is still at an early stage and may exhibit some weirdness and contain bugs
 ## Current Limitations and things to know
 You should be aware of the following:
 
+- Lags are to be expected for networked mount points such as SMB and NFS etc.
+
 - It may seem natural, but the GUI and CLI operate on the same database, so if you add files using the CLI, the GUI will know about them too.
 
 - If a file is listed in the "Search results" tab, it does not imply that a preview will be available in the "Previews" tab, as looqs can search more file formats than it can generate previews for currently.
@@ -26,6 +28,7 @@ Database default path: `$HOME/.local/share/quitesimple.org/looqs/looqs.sqlite`.
 ### First run
 You will be presented with an empty list. Go to the **"Index"** tab, add some directories and click **"Start indexing"**.
 
+### Indexing
 For large directories the progress bar is essentially just decoration. As long as you see the counters
 increase, everything is fine even if it seems the progress bar is stuck.
 
@@ -33,9 +36,17 @@ The indexing can be stopped. If you run it again you do not start from scratch,
 which files have been modified since they have been added to the index. Thus, files will
 only be reprocessed when necessary. Note that cancellation itself may take a moment as files finish processing.
 
+The counters increase in batches, therefore it's normal that it seems no progress is being made, particularly when processing lots of large documents. This aspect will be improved in a future version.
+
 ### Search
 The text field at the top is where you type your query. It can be selected quickly using **CTRL + L**. Filters are available, see this document at the end. By default, both the full path and the content are searched. Path names take precedence, i. e. they will appear the top of the list.
 
+**CTRL + F**: This is helpful shortcut if you want to perform several searches. Consider the following
+query: "p:(docs) c:(invoice credit card)". Press CTRL+F to highlight 'invoice credit card'. This way
+you can quickly perform content searches in paths containing 'docs'.
+
+**CTRL + W**: Removes the last filter. If we take above's example "p:(docs) c:(invoice credit card)" again, then CTRL + W kills "c:(invoice credit card)".
+
 ### Configuring PDF viewer
 It's most convenient if, when you click on a preview, the PDF reader opens the page you clicked. For that, looqs needs to know which viewer you want to launch.
 

gui/mainwindow.cpp

@@ -341,6 +341,102 @@ bool MainWindow::indexerTabActive()
 	return ui->tabWidget->currentIndex() == 2;
 }
 
+void MainWindow::processShortcut(int key)
+{
+	if(key == Qt::Key_Tab || key == Qt::Key_Backtab)
+	{
+		int tabIndex = ui->tabWidget->currentIndex();
+		if(key == Qt::Key_Tab)
+		{
+			++tabIndex;
+		}
+		if(key == Qt::Key_Backtab)
+		{
+			--tabIndex;
+		}
+		tabIndex = tabIndex % ui->tabWidget->count();
+		if(tabIndex < 0)
+		{
+			tabIndex = ui->tabWidget->count() - 1;
+		}
+		ui->tabWidget->setCurrentIndex(tabIndex);
+	}
+	if(key == Qt::Key_L)
+	{
+		ui->txtSearch->setFocus();
+		ui->txtSearch->selectAll();
+	}
+	if(key == Qt::Key_W)
+	{
+		ui->txtSearch->setFocus();
+		QString currentText = ui->txtSearch->text().trimmed();
+		int index = currentText.lastIndexOf(QRegularExpression("[\\s\\)]"));
+		if(index != -1)
+		{
+			bool isFilter = (index == currentText.length() - 1);
+			currentText.remove(index + 1, currentText.length() - index - 1);
+			if(isFilter)
+			{
+				index = currentText.lastIndexOf(' ', index);
+				if(index == -1)
+				{
+					index = 0;
+				}
+				currentText.remove(index, currentText.length());
+			}
+			if(currentText.length() > 0)
+			{
+				currentText += ' ';
+			}
+			ui->txtSearch->setText(currentText);
+		}
+		else
+		{
+			ui->txtSearch->clear();
+		}
+	}
+	if(key == Qt::Key_F)
+	{
+		ui->txtSearch->setFocus();
+		QString currentText = ui->txtSearch->text().trimmed();
+		int index = currentText.lastIndexOf(')');
+		if(index != -1)
+		{
+			bool isFilter = (index == currentText.length() - 1);
+			if(!isFilter)
+			{
+				ui->txtSearch->setSelection(index + 2, ui->txtSearch->text().length() - index - 1);
+			}
+			else
+			{
+				int begin = currentText.lastIndexOf('(', index - 1);
+				if(begin != -1)
+				{
+					ui->txtSearch->setSelection(begin + 1, index - begin - 1);
+				}
+			}
+		}
+		else
+		{
+			int spaceIndex = currentText.lastIndexOf(' ');
+			int colonIndex = currentText.lastIndexOf(':');
+			if(colonIndex > spaceIndex)
+			{
+				int target = currentText.indexOf(' ', colonIndex);
+				if(target == -1)
+				{
+					target = ui->txtSearch->text().size() - colonIndex;
+				}
+				ui->txtSearch->setSelection(colonIndex + 1, target - 1);
+			}
+			else
+			{
+				ui->txtSearch->setSelection(spaceIndex + 1, ui->txtSearch->text().size() - spaceIndex - 1);
+			}
+		}
+	}
+}
+
 void MainWindow::keyPressEvent(QKeyEvent *event)
 {
 	bool quit =
@@ -348,17 +444,15 @@ void MainWindow::keyPressEvent(QKeyEvent *event)
 	if(quit)
 	{
 		qApp->quit();
+		return;
 	}
 
 	if(event->modifiers() & Qt::ControlModifier)
 	{
+		processShortcut(event->key());
+		return;
+	}
 
-		if(event->key() == Qt::Key_L)
-		{
-			ui->txtSearch->setFocus();
-			ui->txtSearch->selectAll();
-		}
-	}
 	QWidget::keyPressEvent(event);
 }
 
@@ -408,6 +502,17 @@ void MainWindow::initSettingsTabs()
 
 void MainWindow::saveSettings()
 {
+	if(ui->chkRemoveOldDb->isChecked())
+	{
+		bool result = QFile::remove(Common::databasePath());
+		if(!result)
+		{
+			QMessageBox::critical(this, "Error removing database",
+								  "Failed to remove old database. Settings not saved.");
+			return;
+		}
+	}
+
 	QSettings settings;
 
 	QString pdfViewerCmd = ui->txtSettingPdfPreviewerCmd->text();
@@ -588,6 +693,8 @@ void MainWindow::lineEditReturnPressed()
 void MainWindow::handleSearchResults(const QVector<SearchResult> &results)
 {
 	this->previewableSearchResults.clear();
+	qDeleteAll(ui->scrollAreaWidgetContents->children());
+
 	ui->treeResultsList->clear();
 	ui->comboPreviewFiles->clear();
 	ui->comboPreviewFiles->addItem("All previews");
@@ -669,7 +776,7 @@ void MainWindow::makePreviews(int page)
 	processedPdfPreviews = 0;
 
 	QVector<QString> wordsToHighlight;
-	QRegularExpression extractor(R"#("([^"]*)"|((\p{L}|\p{N})+))#");
+	QRegularExpression extractor(R"#("([^"]*)"|([^\s]+))#");
 	for(const Token &token : this->contentSearchQuery.getTokens())
 	{
 		if(token.type == FILTER_CONTENT_CONTAINS)

gui/mainwindow.h

@@ -63,6 +63,7 @@ class MainWindow : public QMainWindow
 	unsigned int currentPreviewGeneration = 1;
 	void initSettingsTabs();
 	int currentSelectedScale();
+	void processShortcut(int key);
 private slots:
 	void lineEditReturnPressed();
 	void treeSearchItemActivated(QTreeWidgetItem *item, int i);

gui/mainwindow.ui

@@ -532,6 +532,13 @@
            <item>
             <widget class="QLineEdit" name="txtSettingDatabasePath"/>
            </item>
+           <item>
+            <widget class="QCheckBox" name="chkRemoveOldDb">
+             <property name="text">
+              <string>Remove old database on save</string>
+             </property>
+            </widget>
+           </item>
           </layout>
          </widget>
         </item>

gui/previewgeneratorplaintext.cpp

@@ -2,6 +2,7 @@
 
 #include "previewgeneratorplaintext.h"
 #include "previewresultplaintext.h"
+#include "../shared/limitqueue.h"
 
 QString PreviewGeneratorPlainText::generatePreviewText(QString content, RenderConfig config, QString fileName)
 {
@@ -14,14 +15,13 @@ QString PreviewGeneratorPlainText::generatePreviewText(QString content, RenderCo
 
 	QHash<QString, int> countmap;
 
-	const unsigned int maxSnippets = 7;
 	unsigned int currentSnippets = 0;
 	for(QString &word : config.wordsToHighlight)
 	{
 
 		int lastPos = 0;
 		int index = content.indexOf(word, lastPos, Qt::CaseInsensitive);
-		while(index != -1 && currentSnippets < maxSnippets)
+		while(index != -1 && currentSnippets < MAX_SNIPPETS)
 		{
 			countmap[word] = countmap.value(word, 0) + 1;
 
@@ -57,17 +57,14 @@ QString PreviewGeneratorPlainText::generatePreviewText(QString content, RenderCo
 		++i;
 	}
 
-	for(QString &word : config.wordsToHighlight)
-	{
-		resulText.replace(word, "<span style=\"background-color: yellow;\">" + word + "</span>", Qt::CaseInsensitive);
-	}
-
 	QString header = "<b>" + fileName + "</b> ";
 	for(QString &word : config.wordsToHighlight)
 	{
+		resulText.replace(word, "<span style=\"background-color: yellow;\">" + word + "</span>", Qt::CaseInsensitive);
 		header += word + ": " + QString::number(countmap[word]) + " ";
 	}
-	if(currentSnippets == maxSnippets)
+
+	if(currentSnippets == MAX_SNIPPETS)
 	{
 		header += "(truncated)";
 	}
@@ -77,6 +74,82 @@ QString PreviewGeneratorPlainText::generatePreviewText(QString content, RenderCo
 	return header + resulText.replace("\n", "<br>").mid(0, 1000);
 }
 
+QString PreviewGeneratorPlainText::generateLineBasedPreviewText(QTextStream &in, RenderConfig config, QString fileName)
+{
+	QString resultText;
+	const unsigned int contextLinesCount = 2;
+	LimitQueue<QString> queue(contextLinesCount);
+	QString currentLine;
+	currentLine.reserve(512);
+
+	/* How many lines to read after a line with a match (like grep -A ) */
+	int justReadLinesCount = -1;
+
+	auto appendLine = [&resultText](int lineNumber, QString &line)
+	{ resultText.append(QString("<b>%1</b>%2<br>").arg(lineNumber).arg(line)); };
+
+	QHash<QString, int> countmap;
+	QString header = "<b>" + fileName + "</b> ";
+
+	unsigned int snippetsCount = 0;
+	unsigned int lineCount = 0;
+	while(in.readLineInto(&currentLine) && snippetsCount < MAX_SNIPPETS)
+	{
+		++lineCount;
+		bool matched = false;
+		if(justReadLinesCount > 0)
+		{
+			appendLine(lineCount, currentLine);
+			--justReadLinesCount;
+			continue;
+		}
+		if(justReadLinesCount == 0)
+		{
+			resultText += "---<br>";
+			justReadLinesCount = -1;
+			++snippetsCount;
+		}
+		for(QString &word : config.wordsToHighlight)
+		{
+			if(currentLine.contains(word, Qt::CaseInsensitive))
+			{
+				countmap[word] = countmap.value(word, 0) + 1;
+				matched = true;
+				currentLine.replace(word, "<span style=\"background-color: yellow;\">" + word + "</span>",
+									Qt::CaseInsensitive);
+			}
+		}
+		if(matched)
+		{
+			while(queue.size() > 0)
+			{
+				int queuedLineCount = lineCount - queue.size();
+				QString queuedLine = queue.dequeue();
+
+				appendLine(queuedLineCount, queuedLine);
+			}
+			appendLine(lineCount, currentLine);
+			justReadLinesCount = contextLinesCount;
+		}
+		else
+		{
+			queue.enqueue(currentLine);
+		}
+	}
+
+	for(QString &word : config.wordsToHighlight)
+	{
+		header += word + ": " + QString::number(countmap[word]) + " ";
+	}
+	if(snippetsCount == MAX_SNIPPETS)
+	{
+		header += "(truncated)";
+	}
+	header += "<hr>";
+
+	return header + resultText;
+}
+
 QSharedPointer<PreviewResult> PreviewGeneratorPlainText::generate(RenderConfig config, QString documentPath,
 																  unsigned int page)
 {
@@ -87,9 +160,7 @@ QSharedPointer<PreviewResult> PreviewGeneratorPlainText::generate(RenderConfig c
 		return QSharedPointer<PreviewResultPlainText>(result);
 	}
 	QTextStream in(&file);
-
-	QString content = in.readAll();
 	QFileInfo info{documentPath};
-	result->setText(generatePreviewText(content, config, info.fileName()));
+	result->setText(generateLineBasedPreviewText(in, config, info.fileName()));
 	return QSharedPointer<PreviewResultPlainText>(result);
 }

gui/previewgeneratorplaintext.h

@@ -1,12 +1,17 @@
 #ifndef PREVIEWGENERATORPLAINTEXT_H
 #define PREVIEWGENERATORPLAINTEXT_H
+#include <QTextStream>
 #include "previewgenerator.h"
 
 class PreviewGeneratorPlainText : public PreviewGenerator
 {
+  protected:
+	const unsigned int MAX_SNIPPETS = 7;
+
   public:
 	using PreviewGenerator::PreviewGenerator;
 	QString generatePreviewText(QString content, RenderConfig config, QString fileName);
+	QString generateLineBasedPreviewText(QTextStream &in, RenderConfig config, QString fileName);
 	QSharedPointer<PreviewResult> generate(RenderConfig config, QString documentPath, unsigned int page);
 };
 

shared/limitqueue.h

@@ -0,0 +1,48 @@
+#ifndef LIMITQUEUE_H
+#define LIMITQUEUE_H
+#include <QQueue>
+
+template <class T> class LimitQueue
+{
+  protected:
+	QQueue<T> queue;
+	unsigned int limit = 0;
+
+  public:
+	LimitQueue();
+	LimitQueue(unsigned int limit)
+	{
+		this->limit = limit;
+	}
+
+	void enqueue(const T &t)
+	{
+		if(queue.size() == limit)
+		{
+			queue.dequeue();
+		}
+		queue.enqueue(t);
+	}
+
+	int size()
+	{
+		return queue.size();
+	}
+
+	T dequeue()
+	{
+		return queue.dequeue();
+	}
+
+	void setLimit(unsigned int limit)
+	{
+		this->limit = limit;
+	}
+
+	void clear()
+	{
+		queue.clear();
+	}
+};
+
+#endif // LIMITQUEUE_H

shared/looqsquery.cpp

@@ -180,9 +180,8 @@ LooqsQuery LooqsQuery::build(QString expression, TokenType loneWordsTokenType, b
 
 	QStringList loneWords;
 	LooqsQuery result;
-	QRegularExpression rx(
+	QRegularExpression rx("((?<filtername>(\\.|\\w)+):(?<args>\\((?<innerargs>[^\\)]+)\\)|([^\\s])+)|(?<boolean>AND|OR)"
-		"((?<filtername>(\\.|\\w)+):(?<args>\\((?<innerargs>[^\\)]+)\\)|([\\p{L}\\p{N},])+)|(?<boolean>AND|OR)"
+						  "|(?<negation>!)|(?<bracket>\\(|\\))|(?<loneword>[^\\s]+))");
-		"|(?<negation>!)|(?<bracket>\\(|\\))|(?<loneword>[\"\\p{L}\\p{N}]+))");
 	QRegularExpressionMatchIterator i = rx.globalMatch(expression);
 	auto previousWasBool = [&result] { return !result.tokens.empty() && ((result.tokens.last().type & BOOL) == BOOL); };
 	auto previousWas = [&result](TokenType t) { return !result.tokens.empty() && (result.tokens.last().type == t); };

shared/migrations/3.sql

@@ -0,0 +1 @@
+DROP trigger content_ad;

shared/migrations/migrations.qrc

@@ -2,5 +2,6 @@
     <qresource prefix="/looqs-migrations">
         <file>1.sql</file>
         <file>2.sql</file>
+        <file>3.sql</file>
     </qresource>
 </RCC>

shared/shared.pro

@@ -77,6 +77,7 @@ HEADERS += sqlitesearch.h \
     filescanworker.h \
     indexer.h \
     indexsyncer.h \
+    limitqueue.h \
     logger.h \
     looqsgeneralexception.h \
     looqsquery.h \