18 次代码提交
v0.7 ... v0.8.1

作者 SHA1 备注 提交日期
20a1f8b2cd Release v0.8.1 2022-11-19 11:54:24 +01:00
a47af257f3 gui: previews: Fix incorrect pos calculation in cached previews
The cached order position introduced in 42e9ac5 is incorrect
as it does not consider the case when we are viewing any
other result page than the first.

Fix this by considering which page we are in when calculating
the offset
2022-11-18 22:22:11 +01:00
9686ef30c7 gui: PreviewResult*: Wrap result in shared pointer immediatly 2022-11-13 17:37:35 +01:00
abce4cfcd9 gui: PreviewGeneratorPlaintext: Escape words we pass to QRegularExpression 2022-11-13 17:27:45 +01:00
d55187a71c submodules: exile.h: Sync to current master 2022-10-26 13:14:03 +02:00
9e1bc98f38 gui: mainwindow.h: Initialize preview-related members with a default 2022-10-26 13:13:20 +02:00
496aefaa09 shared: sqlitesearch: Remove unused var 2022-10-26 13:10:00 +02:00
b4320f611b Release v0.8, minor README changes 2022-10-22 17:20:20 +02:00
1b1ab2387e gui: PreviewGeneratorPdf: Guard cache lookup with mutex
No guarantes the read-only lookup is thread-safe so better
just lock there too
2022-10-22 15:08:29 +02:00
49a1a14009 gui: previewgenerator: Use QHash and guard using mutexes 2022-10-22 15:07:46 +02:00
48ca25abe3 gui: mainwindow: Reorder members for readability 2022-10-19 11:56:21 +02:00
42e9ac5f41 gui: previews: Ensure order matches relevance ranking
Previously, the order of previews would depend simply
on which generator would finish first.

Fix this by caching out of order previews. This may
cause a small delay but should overall be hardly noticable.
2022-10-19 11:11:29 +02:00
7d3c24e6e1 update README.md,HACKING.md 2022-10-18 16:19:25 +02:00
c155d25a37 shared: sqlitesearch: Search trigram index too
Search the trigram index too, combining the results
with the results of the "normal" fts index.

Prioritize the latter since it makes more sense to
rank whole words higher.
2022-10-18 16:06:10 +02:00
583d5babf3 shared: sqlitedbservice: Insert to trigram index too 2022-10-18 16:05:19 +02:00
45659cdc59 shared: migrations: Add 4.sql: Begin trigram index 2022-10-18 16:04:00 +02:00
3022bbdfb5 sqlitesearch: escapeftsArgument: Fix wrong escaping of phrase queries 2022-10-02 19:55:10 +02:00
b6ac652ade shared: indexer: Report progress more often
Processing dirs with large docs takes time and waiting till the threshold
is reached can be a bit annoying in those cases, so report if last report
was 10+ seconds ago.
2022-09-23 20:08:00 +02:00
共有 18 个文件被更改,包括 183 次插入93 次删除

查看文件

@ -1,4 +1,21 @@
# looqs: Release notes # looqs: Release notes
## 2022-11-19 - v0.8.1
CHANGES:
- Fix regression causing previews in second (and higher) result page to not render
- Minor improvements
## 2022-10-22 - v0.8
CHANGES:
- For new, not previously indexed files, start creating an additional index using sqlite's experimental trigram tokenizer. Thanks to that, we can now match substrings >= 3 of an unicode sequence. Results of the usual index are prioritized.
- GUI: Ensure order of previews matches ranking exactly. Previously, it depended simply on the time preview generators took, i. e. it was more or less a race.
- Report progress more often during indexing, so users don't get the impression that it's stuck when processing dirs with large documents.
- Fix a regression that caused phrase queries to be broken
- Minor improvements
- Add packages: Ubuntu 22.10.
## 2022-09-10 - v0.7 ## 2022-09-10 - v0.7
CHANGES: CHANGES:

查看文件

@ -1,12 +1,7 @@
# looqs - Hacking # looqs - Hacking
## Introduction ## Introduction
Without elaborating here, I hacked looqs because I was not satisfied with the state of desktop search on Linux. If you are interested on how to contribute, please see the file [CONTRIBUTING.md](CONTRIBUTING.md) which contains the instructions on how to submit patches etc.
Originally a set of CLI python scripts, it is now written in C++ and offers a GUI made using Qt. While a "web app" would have been an option, I prefer a desktop application for something like looqs. I chose Qt because I am more familiar with it than with any other GUI framework. To my knowledge, potential alternatives like GTK do not include as many "batteries" as Qt anyway, so the job presumably would have been harder there,
at least for me.
If you are interested in how to contribute, please see the file [CONTRIBUTING.md](CONTRIBUTING.md) which contains the instructions on how to submit patches etc.
## Security ## Security
The architecture ensures that the parsing of documents and the preview generation is sandboxed by [exile.h](https://github.com/quitesimpleorg/exile.h). looqs uses a multi-process architecture to achieve this. The architecture ensures that the parsing of documents and the preview generation is sandboxed by [exile.h](https://github.com/quitesimpleorg/exile.h). looqs uses a multi-process architecture to achieve this.
@ -16,7 +11,8 @@ Qt code is considered trusted in this model. While one may critize this, it was
Set the enviornment variable `LOOQS_DISABLE_SANDBOX=1` to disable sandboxing. It's intended for troublehshooting. Set the enviornment variable `LOOQS_DISABLE_SANDBOX=1` to disable sandboxing. It's intended for troublehshooting.
## Database ## Database
The heart is sqlite, with the FTS5 extensions behind the full-text search. While FTS may not be sqlite's strong suit, I definitly did not want to run one of those oftenly recommended heavy (Java based) solutions. I explored other options like Postgresql, I've discard them due to some limitations back then. The heart is sqlite, with the FTS5 extensions behind the full-text search. While FTS may not be sqlite's strong suit, I definitely did not want to run one of those oftenly recommended heavy (Java based) solutions. I explored other options like Postgresql, I've discard them due to some limitations back then. It's also natural to use sqlite as it's
used for metadata in general.
Down the road, alternatives will be explored of course if sqlite should not suffice anymore. Down the road, alternatives will be explored of course if sqlite should not suffice anymore.
@ -27,7 +23,7 @@ looqs simply strips the tags and that seems to work fine so far. Naturally, this
Naturally, looqs won't be able to index and render previews for everything. Such approach would create a huge bloated binary. In the future, there will be some plugin system of some sorts, either we will load .so objects or use subprocesses. Naturally, looqs won't be able to index and render previews for everything. Such approach would create a huge bloated binary. In the future, there will be some plugin system of some sorts, either we will load .so objects or use subprocesses.
## Name ## Name
looqs looks for files. You as the user can also look inside them. The 'k' in "looks" was replaced by a 'q'. Originally, I wanted my projects to have "qs" (for quitesimple) in their name. While abandoned now, this got us to looqs. looqs looks for files. You as the user can also look inside them. The 'k' in "looks" was replaced by a 'q'. Originally, I wanted my projects to have "qs" (for quitesimple) in their name. While that quirk is abandoned now, this got us to looqs.

查看文件

@ -28,9 +28,12 @@ There is no need to write the long form of filters. There are also booleans avai
The screenshots in this section may occasionally be slightly outdated, but they are usually recent enough to get an overall impression of the current state of the GUI. The screenshots in this section may occasionally be slightly outdated, but they are usually recent enough to get an overall impression of the current state of the GUI.
## Current status ## Current status
Latest version: 2022-09-10, v0.7 Latest version: 2022-11-19, v0.8.1
Please see [Changelog](CHANGELOG.md) for a human readable list of changes. Please keep in mind: looqs is still at an early stage and may exhibit some weirdness and contain bugs.
Please see [Changelog](CHANGELOG.md) for a human readable list of changes. For download instructions, see
further down this document.
## Goals and principles ## Goals and principles
@ -94,7 +97,7 @@ The GUI is located in `gui/looqs-gui`, the binary for the CLI is in `cli/looqs`
## Packages ## Packages
At this point, looqs is not in any official distro package repo, but I maintain some packages. At this point, looqs is not in any official distro package repo, but I maintain some packages.
### Ubuntu 22.04 ### Ubuntu 22.04, 22.10
Latest release can be installed using apt from the repo. Latest release can be installed using apt from the repo.
``` ```
# First, obtain key, assume it's trusted. # First, obtain key, assume it's trusted.

查看文件

@ -643,7 +643,6 @@ void MainWindow::previewReceived(QSharedPointer<PreviewResult> preview, unsigned
{ {
QString docPath = preview->getDocumentPath(); QString docPath = preview->getDocumentPath();
auto previewPage = preview->getPage(); auto previewPage = preview->getPage();
ClickLabel *headerLabel = new ClickLabel(); ClickLabel *headerLabel = new ClickLabel();
headerLabel->setText(QString("Path: ") + preview->getDocumentPath()); headerLabel->setText(QString("Path: ") + preview->getDocumentPath());
@ -685,7 +684,24 @@ void MainWindow::previewReceived(QSharedPointer<PreviewResult> preview, unsigned
previewWidget->setLayout(previewLayout); previewWidget->setLayout(previewLayout);
ui->scrollAreaWidgetContents->layout()->addWidget(previewWidget); QBoxLayout *layout = static_cast<QBoxLayout *>(ui->scrollAreaWidgetContents->layout());
int pos = previewOrder[docPath + QString::number(previewPage)];
if(pos <= layout->count())
{
layout->insertWidget(pos, previewWidget);
for(auto it = previewWidgetOrderCache.constKeyValueBegin();
it != previewWidgetOrderCache.constKeyValueEnd(); it++)
{
if(it->first <= layout->count())
{
layout->insertWidget(it->first, it->second);
}
}
}
else
{
previewWidgetOrderCache[pos] = previewWidget;
}
} }
} }
@ -924,12 +940,12 @@ void MainWindow::makePreviews(int page)
} }
} }
} }
int end = previewsPerPage; int length = previewsPerPage;
int begin = page * previewsPerPage - previewsPerPage; int beginOffset = page * previewsPerPage - previewsPerPage;
if(begin < 0) if(beginOffset < 0)
{ {
// Should not happen actually // Should not happen actually
begin = 0; beginOffset = 0;
} }
int currentScale = currentSelectedScale(); int currentScale = currentSelectedScale();
@ -938,6 +954,10 @@ void MainWindow::makePreviews(int page)
renderConfig.scaleY = QGuiApplication::primaryScreen()->physicalDotsPerInchY() * (currentScale / 100.); renderConfig.scaleY = QGuiApplication::primaryScreen()->physicalDotsPerInchY() * (currentScale / 100.);
renderConfig.wordsToHighlight = wordsToHighlight; renderConfig.wordsToHighlight = wordsToHighlight;
this->previewOrder.clear();
this->previewWidgetOrderCache.clear();
int previewPos = 0;
QVector<RenderTarget> targets; QVector<RenderTarget> targets;
for(SearchResult &sr : this->previewableSearchResults) for(SearchResult &sr : this->previewableSearchResults)
{ {
@ -952,10 +972,14 @@ void MainWindow::makePreviews(int page)
renderTarget.path = sr.fileData.absPath; renderTarget.path = sr.fileData.absPath;
renderTarget.page = (int)sr.page; renderTarget.page = (int)sr.page;
targets.append(renderTarget); targets.append(renderTarget);
int pos = previewPos - beginOffset;
this->previewOrder[renderTarget.path + QString::number(renderTarget.page)] = pos;
++previewPos;
} }
int numpages = ceil(static_cast<double>(targets.size()) / previewsPerPage); int numpages = ceil(static_cast<double>(targets.size()) / previewsPerPage);
ui->spinPreviewPage->setMaximum(numpages); ui->spinPreviewPage->setMaximum(numpages);
targets = targets.mid(begin, end); targets = targets.mid(beginOffset, length);
ui->lblTotalPreviewPagesCount->setText(QString::number(numpages)); ui->lblTotalPreviewPagesCount->setText(QString::number(numpages));
ui->previewProcessBar->setMaximum(targets.count()); ui->previewProcessBar->setMaximum(targets.count());

查看文件

@ -23,16 +23,6 @@ class MainWindow : public QMainWindow
{ {
Q_OBJECT Q_OBJECT
public:
explicit MainWindow(QWidget *parent, QString socketPath);
~MainWindow();
signals:
void beginSearch(const QString &query);
void startPdfPreviewGeneration(QVector<SearchResult> paths, double scalefactor);
protected:
void closeEvent(QCloseEvent *event) override;
private: private:
DatabaseFactory *dbFactory; DatabaseFactory *dbFactory;
SqliteDbService *dbService; SqliteDbService *dbService;
@ -40,37 +30,40 @@ class MainWindow : public QMainWindow
IPCPreviewClient ipcPreviewClient; IPCPreviewClient ipcPreviewClient;
QThread ipcClientThread; QThread ipcClientThread;
QThread syncerThread; QThread syncerThread;
Indexer *indexer;
IndexSyncer *indexSyncer; IndexSyncer *indexSyncer;
QProgressDialog progressDialog; QProgressDialog progressDialog;
Indexer *indexer;
QFileIconProvider iconProvider; QFileIconProvider iconProvider;
bool previewDirty;
QSqlDatabase db; QSqlDatabase db;
QFutureWatcher<QVector<SearchResult>> searchWatcher; QFutureWatcher<QVector<SearchResult>> searchWatcher;
void add(QString path, unsigned int page);
QVector<SearchResult> previewableSearchResults; QVector<SearchResult> previewableSearchResults;
LooqsQuery contentSearchQuery;
QVector<QString> searchHistory;
int currentSearchHistoryIndex = 0;
QString currentSavedSearchText;
QHash<QString, int> previewOrder; /* Quick lookup for the order a preview should have */
QMap<int, QWidget *>
previewWidgetOrderCache /* Saves those that arrived out of order to be inserted later at the correct pos */;
bool previewDirty = false;
int previewsPerPage = 20;
unsigned int processedPdfPreviews = 0;
unsigned int currentPreviewGeneration = 1;
void connectSignals(); void connectSignals();
void makePreviews(int page); void makePreviews(int page);
bool previewTabActive(); bool previewTabActive();
bool indexerTabActive(); bool indexerTabActive();
void keyPressEvent(QKeyEvent *event) override; void keyPressEvent(QKeyEvent *event) override;
unsigned int processedPdfPreviews;
void handleSearchResults(const QVector<SearchResult> &results); void handleSearchResults(const QVector<SearchResult> &results);
void handleSearchError(QString error); void handleSearchError(QString error);
LooqsQuery contentSearchQuery;
int previewsPerPage;
void createSearchResutlMenu(QMenu &menu, const QFileInfo &fileInfo); void createSearchResutlMenu(QMenu &menu, const QFileInfo &fileInfo);
void openDocument(QString path, int num); void openDocument(QString path, int num);
void openFile(QString path); void openFile(QString path);
unsigned int currentPreviewGeneration = 1;
void initSettingsTabs(); void initSettingsTabs();
int currentSelectedScale(); int currentSelectedScale();
void processShortcut(int key); void processShortcut(int key);
bool eventFilter(QObject *object, QEvent *event); bool eventFilter(QObject *object, QEvent *event);
QVector<QString> searchHistory;
int currentSearchHistoryIndex = 0;
QString currentSavedSearchText;
private slots: private slots:
void lineEditReturnPressed(); void lineEditReturnPressed();
void treeSearchItemActivated(QTreeWidgetItem *item, int i); void treeSearchItemActivated(QTreeWidgetItem *item, int i);
@ -90,6 +83,16 @@ class MainWindow : public QMainWindow
void startIpcPreviews(RenderConfig config, const QVector<RenderTarget> &targets); void startIpcPreviews(RenderConfig config, const QVector<RenderTarget> &targets);
void stopIpcPreviews(); void stopIpcPreviews();
void beginIndexSync(); void beginIndexSync();
public:
explicit MainWindow(QWidget *parent, QString socketPath);
~MainWindow();
signals:
void beginSearch(const QString &query);
void startPdfPreviewGeneration(QVector<SearchResult> paths, double scalefactor);
protected:
void closeEvent(QCloseEvent *event) override;
}; };
#endif // MAINWINDOW_H #endif // MAINWINDOW_H

查看文件

@ -1,20 +1,24 @@
#include "../shared/common.h" #include "../shared/common.h"
#include "previewgenerator.h" #include "previewgenerator.h"
#include <QMutexLocker>
#include "previewgeneratorpdf.h" #include "previewgeneratorpdf.h"
#include "previewgeneratorplaintext.h" #include "previewgeneratorplaintext.h"
#include "previewgeneratorodt.h" #include "previewgeneratorodt.h"
static PreviewGenerator *plainTextGenerator = new PreviewGeneratorPlainText(); static PreviewGenerator *plainTextGenerator = new PreviewGeneratorPlainText();
static QMap<QString, PreviewGenerator *> generators{ static QHash<QString, PreviewGenerator *> generators{
{"pdf", new PreviewGeneratorPdf()}, {"txt", plainTextGenerator}, {"md", plainTextGenerator}, {"pdf", new PreviewGeneratorPdf()}, {"txt", plainTextGenerator}, {"md", plainTextGenerator},
{"py", plainTextGenerator}, {"java", plainTextGenerator}, {"js", plainTextGenerator}, {"py", plainTextGenerator}, {"java", plainTextGenerator}, {"js", plainTextGenerator},
{"cpp", plainTextGenerator}, {"c", plainTextGenerator}, {"sql", plainTextGenerator}, {"cpp", plainTextGenerator}, {"c", plainTextGenerator}, {"sql", plainTextGenerator},
{"odt", new PreviewGeneratorOdt()}}; {"odt", new PreviewGeneratorOdt()}};
static QMutex generatorsMutex;
PreviewGenerator *PreviewGenerator::get(QFileInfo &info) PreviewGenerator *PreviewGenerator::get(QFileInfo &info)
{ {
QMutexLocker locker(&generatorsMutex);
PreviewGenerator *result = generators.value(info.suffix(), nullptr); PreviewGenerator *result = generators.value(info.suffix(), nullptr);
locker.unlock();
if(result == nullptr) if(result == nullptr)
{ {
if(Common::isTextFile(info)) if(Common::isTextFile(info))

查看文件

@ -7,10 +7,12 @@ static QMutex cacheMutex;
Poppler::Document *PreviewGeneratorPdf::document(QString path) Poppler::Document *PreviewGeneratorPdf::document(QString path)
{ {
QMutexLocker locker(&cacheMutex);
if(documentcache.contains(path)) if(documentcache.contains(path))
{ {
return documentcache.value(path); return documentcache.value(path);
} }
locker.unlock();
Poppler::Document *result = Poppler::Document::load(path); Poppler::Document *result = Poppler::Document::load(path);
if(result == nullptr) if(result == nullptr)
{ {
@ -19,7 +21,7 @@ Poppler::Document *PreviewGeneratorPdf::document(QString path)
} }
result->setRenderHint(Poppler::Document::TextAntialiasing); result->setRenderHint(Poppler::Document::TextAntialiasing);
QMutexLocker locker(&cacheMutex); locker.relock();
documentcache.insert(path, result); documentcache.insert(path, result);
locker.unlock(); locker.unlock();
return result; return result;

查看文件

@ -103,7 +103,7 @@ QString PreviewGeneratorPlainText::generateLineBasedPreviewText(QTextStream &in,
int foundWordsCount = 0; int foundWordsCount = 0;
for(QString &word : config.wordsToHighlight) for(QString &word : config.wordsToHighlight)
{ {
QRegularExpression searchRegex("\\b" + word + "\\b"); QRegularExpression searchRegex("\\b" + QRegularExpression::escape(word) + "\\b");
bool containsRegex = line.contains(searchRegex); bool containsRegex = line.contains(searchRegex);
bool contains = false; bool contains = false;
if(!containsRegex) if(!containsRegex)

查看文件

@ -30,7 +30,7 @@ QByteArray PreviewResultPdf::serialize() const
QSharedPointer<PreviewResultPdf> PreviewResultPdf::deserialize(QByteArray &ba) QSharedPointer<PreviewResultPdf> PreviewResultPdf::deserialize(QByteArray &ba)
{ {
PreviewResultPdf *result = new PreviewResultPdf(); QSharedPointer<PreviewResultPdf> result(new PreviewResultPdf());
PreviewResultType type; PreviewResultType type;
QDataStream stream{&ba, QIODevice::ReadOnly}; QDataStream stream{&ba, QIODevice::ReadOnly};
@ -40,5 +40,5 @@ QSharedPointer<PreviewResultPdf> PreviewResultPdf::deserialize(QByteArray &ba)
throw std::runtime_error("Invalid byte array: Not a pdf preview"); throw std::runtime_error("Invalid byte array: Not a pdf preview");
} }
stream >> result->documentPath >> result->page >> result->previewImage; stream >> result->documentPath >> result->page >> result->previewImage;
return QSharedPointer<PreviewResultPdf>(result); return result;
} }

查看文件

@ -40,7 +40,8 @@ QByteArray PreviewResultPlainText::serialize() const
QSharedPointer<PreviewResultPlainText> PreviewResultPlainText::deserialize(QByteArray &ba) QSharedPointer<PreviewResultPlainText> PreviewResultPlainText::deserialize(QByteArray &ba)
{ {
PreviewResultPlainText *result = new PreviewResultPlainText(); QSharedPointer<PreviewResultPlainText> result(new PreviewResultPlainText());
PreviewResultType type; PreviewResultType type;
QDataStream stream{&ba, QIODevice::ReadOnly}; QDataStream stream{&ba, QIODevice::ReadOnly};
@ -50,5 +51,5 @@ QSharedPointer<PreviewResultPlainText> PreviewResultPlainText::deserialize(QByte
throw std::runtime_error("Invalid byte array: Not a pdf preview"); throw std::runtime_error("Invalid byte array: Not a pdf preview");
} }
stream >> result->documentPath >> result->page >> result->text; stream >> result->documentPath >> result->page >> result->text;
return QSharedPointer<PreviewResultPlainText>(result); return result;
} }

查看文件

@ -152,12 +152,14 @@ void Indexer::processFileScanResult(FileScanResult result)
++this->currentIndexResult.erroredPaths; ++this->currentIndexResult.erroredPaths;
} }
if(currentScanProcessedCount++ == progressReportThreshold) QTime currentTime = QTime::currentTime();
if(currentScanProcessedCount++ == progressReportThreshold || this->lastProgressReportTime.secsTo(currentTime) >= 10)
{ {
emit indexProgress(this->currentIndexResult.total(), this->currentIndexResult.addedPaths, emit indexProgress(this->currentIndexResult.total(), this->currentIndexResult.addedPaths,
this->currentIndexResult.skippedPaths, this->currentIndexResult.erroredPaths, this->currentIndexResult.skippedPaths, this->currentIndexResult.erroredPaths,
this->dirScanner->pathCount()); this->dirScanner->pathCount());
currentScanProcessedCount = 0; currentScanProcessedCount = 0;
this->lastProgressReportTime = currentTime;
} }
} }

查看文件

@ -72,6 +72,8 @@ class Indexer : public QObject
IndexResult currentIndexResult; IndexResult currentIndexResult;
void launchWorker(ConcurrentQueue<QString> &queue, int batchsize); void launchWorker(ConcurrentQueue<QString> &queue, int batchsize);
QTime lastProgressReportTime = QTime::currentTime();
public: public:
bool isRunning(); bool isRunning();

3
shared/migrations/4.sql 普通文件
查看文件

@ -0,0 +1,3 @@
CREATE VIRTUAL TABLE fts_trigram USING fts5(content, content='',tokenize="trigram");
ALTER TABLE content ADD COLUMN fts_trigramid integer;
CREATE INDEX content_fts_trigramid ON content (fts_trigramid);

查看文件

@ -3,5 +3,6 @@
<file>1.sql</file> <file>1.sql</file>
<file>2.sql</file> <file>2.sql</file>
<file>3.sql</file> <file>3.sql</file>
<file>4.sql</file>
</qresource> </qresource>
</RCC> </RCC>

查看文件

@ -110,6 +110,44 @@ unsigned int SqliteDbService::getFiles(QVector<FileData> &results, QString wildC
return processedRows; return processedRows;
} }
bool SqliteDbService::insertToFTS(bool useTrigrams, QSqlDatabase &db, int fileid, QVector<PageData> &pageData)
{
QString ftsInsertStatement;
QString contentInsertStatement;
if(useTrigrams)
{
ftsInsertStatement = "INSERT INTO fts_trigram(content) VALUES(?)";
contentInsertStatement = "INSERT INTO content(fileid, page, fts_trigramid) VALUES(?, ?, last_insert_rowid())";
}
else
{
ftsInsertStatement = "INSERT INTO fts(content) VALUES(?)";
contentInsertStatement = "INSERT INTO content(fileid, page, ftsid) VALUES(?, ?, last_insert_rowid())";
}
for(const PageData &data : pageData)
{
QSqlQuery ftsQuery(db);
ftsQuery.prepare(ftsInsertStatement);
ftsQuery.addBindValue(data.content);
if(!ftsQuery.exec())
{
Logger::error() << "Failed fts insertion " << ftsQuery.lastError() << Qt::endl;
return false;
}
QSqlQuery contentQuery(db);
contentQuery.prepare(contentInsertStatement);
contentQuery.addBindValue(fileid);
contentQuery.addBindValue(data.pagenumber);
if(!contentQuery.exec())
{
Logger::error() << "Failed content insertion " << contentQuery.lastError() << Qt::endl;
return false;
}
}
return true;
}
SaveFileResult SqliteDbService::saveFile(QFileInfo fileInfo, QVector<PageData> &pageData) SaveFileResult SqliteDbService::saveFile(QFileInfo fileInfo, QVector<PageData> &pageData)
{ {
QString absPath = fileInfo.absoluteFilePath(); QString absPath = fileInfo.absoluteFilePath();
@ -149,24 +187,18 @@ SaveFileResult SqliteDbService::saveFile(QFileInfo fileInfo, QVector<PageData> &
} }
int lastid = inserterQuery.lastInsertId().toInt(); int lastid = inserterQuery.lastInsertId().toInt();
for(const PageData &data : pageData) if(!insertToFTS(false, db, lastid, pageData))
{ {
QSqlQuery ftsQuery(db); db.rollback();
ftsQuery.prepare("INSERT INTO fts(content) VALUES(?)"); Logger::error() << "Failed to insert data to FTS index " << Qt::endl;
ftsQuery.addBindValue(data.content); return DBFAIL;
ftsQuery.exec(); }
QSqlQuery contentQuery(db); if(!insertToFTS(true, db, lastid, pageData))
contentQuery.prepare("INSERT INTO content(fileid, page, ftsid) VALUES(?, ?, last_insert_rowid())"); {
contentQuery.addBindValue(lastid); db.rollback();
contentQuery.addBindValue(data.pagenumber); Logger::error() << "Failed to insert data to FTS index " << Qt::endl;
if(!contentQuery.exec()) return DBFAIL;
{
db.rollback();
Logger::error() << "Failed content insertion " << contentQuery.lastError() << Qt::endl;
return DBFAIL;
}
} }
if(!db.commit()) if(!db.commit())
{ {
db.rollback(); db.rollback();

查看文件

@ -13,6 +13,7 @@ class SqliteDbService
{ {
private: private:
DatabaseFactory *dbFactory = nullptr; DatabaseFactory *dbFactory = nullptr;
bool insertToFTS(bool useTrigrams, QSqlDatabase &db, int fileid, QVector<PageData> &pageData);
public: public:
SqliteDbService(DatabaseFactory &dbFactory); SqliteDbService(DatabaseFactory &dbFactory);

查看文件

@ -82,11 +82,10 @@ QString SqliteSearch::escapeFtsArgument(QString ftsArg)
{ {
value = value.mid(0, value.size() - 1); value = value.mid(0, value.size() - 1);
} }
result += "\"" + value + "\"*"; result += "\"" + value + "\"* ";
} }
else else
{ {
value = "\"\"" + value + "\"\"";
result += "\"" + value + "\" "; result += "\"" + value + "\" ";
} }
} }
@ -142,9 +141,7 @@ QPair<QString, QVector<QString>> SqliteSearch::createSql(const Token &token)
} }
if(token.type == FILTER_CONTENT_CONTAINS) if(token.type == FILTER_CONTENT_CONTAINS)
{ {
return {" content.id IN (SELECT fts.ROWID FROM fts WHERE fts.content MATCH ? ORDER BY " return {" fts MATCH ? ", {escapeFtsArgument(value)}};
"rank) ",
{escapeFtsArgument(value)}};
} }
throw LooqsGeneralException("Unknown token passed (should not happen)"); throw LooqsGeneralException("Unknown token passed (should not happen)");
} }
@ -160,30 +157,17 @@ QSqlQuery SqliteSearch::makeSqlQuery(const LooqsQuery &query)
throw LooqsGeneralException("Nothing to search for supplied"); throw LooqsGeneralException("Nothing to search for supplied");
} }
bool ftsAlreadyJoined = false;
auto tokens = query.getTokens(); auto tokens = query.getTokens();
for(const Token &token : tokens) for(const Token &token : tokens)
{ {
if(token.type == FILTER_CONTENT_CONTAINS) auto sql = createSql(token);
{ whereSql += sql.first;
if(!ftsAlreadyJoined) bindValues.append(sql.second);
{
joinSql += " INNER JOIN fts ON content.ftsid = fts.ROWID ";
ftsAlreadyJoined = true;
}
whereSql += " fts.content MATCH ? ";
bindValues.append(escapeFtsArgument(token.value));
}
else
{
auto sql = createSql(token);
whereSql += sql.first;
bindValues.append(sql.second);
}
} }
QString prepSql; QString prepSql;
QString sortSql = createSortSql(query.getSortConditions()); QString sortSql = createSortSql(query.getSortConditions());
int bindIterations = 1;
if(isContentSearch) if(isContentSearch)
{ {
if(sortSql.isEmpty()) if(sortSql.isEmpty())
@ -191,12 +175,24 @@ QSqlQuery SqliteSearch::makeSqlQuery(const LooqsQuery &query)
if(std::find_if(tokens.begin(), tokens.end(), if(std::find_if(tokens.begin(), tokens.end(),
[](const Token &t) -> bool { return t.type == FILTER_CONTENT_CONTAINS; }) != tokens.end()) [](const Token &t) -> bool { return t.type == FILTER_CONTENT_CONTAINS; }) != tokens.end())
{ {
sortSql = "ORDER BY rank"; sortSql = "ORDER BY prio, rank";
} }
} }
prepSql = "SELECT file.path AS path, content.page AS page, file.mtime AS mtime, file.size AS size, " QString whereSqlTrigram = whereSql;
"file.filetype AS filetype FROM file INNER JOIN content ON file.id = content.fileid " + whereSqlTrigram.replace("fts MATCH", "fts_trigram MATCH"); // A bit dirty...
joinSql + " WHERE 1=1 AND " + whereSql + " " + sortSql; prepSql =
"SELECT DISTINCT path, page, mtime, size, filetype FROM ("
"SELECT file.path AS path, content.page AS page, file.mtime AS mtime, file.size AS size, "
"file.filetype AS filetype, 0 AS prio, fts.rank AS rank FROM file INNER JOIN content ON file.id = "
"content.fileid "
"INNER JOIN fts ON content.ftsid = fts.ROWID WHERE 1=1 AND " +
whereSql +
"UNION ALL SELECT file.path AS path, content.page AS page, file.mtime AS mtime, file.size AS size, "
"file.filetype AS filetype, 1 as prio, fts_trigram.rank AS rank FROM file INNER JOIN content ON file.id = "
"content.fileid " +
"INNER JOIN fts_trigram ON content.fts_trigramid = fts_trigram.ROWID WHERE 1=1 AND " + whereSqlTrigram +
" ) " + sortSql;
++bindIterations;
} }
else else
{ {
@ -216,11 +212,14 @@ QSqlQuery SqliteSearch::makeSqlQuery(const LooqsQuery &query)
QSqlQuery dbquery(*db); QSqlQuery dbquery(*db);
dbquery.prepare(prepSql); dbquery.prepare(prepSql);
for(const QString &value : bindValues) for(int i = 0; i < bindIterations; i++)
{ {
if(value != "") for(const QString &value : bindValues)
{ {
dbquery.addBindValue(value); if(value != "")
{
dbquery.addBindValue(value);
}
} }
} }
return dbquery; return dbquery;