Commit Graph

829 Commits

Author SHA1 Message Date
Robin Appelman e75797ad67
increase log level for storage self-test
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-06-08 18:12:49 +02:00
Roeland Jago Douma 6aa6ab3e02
Add lazy events for the Node API
Right now if you want to get events via the Node API you have to have a
real instance of the Root. Which in turns sets up the whole FS.

We should make sure this is done lazy. Else enabling the preview
generator for example makes you setup the whole FS on each and every
authenticated call.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-05-28 12:35:45 +02:00
Roeland Jago Douma 6b26744787
Clear the statscache before fetching the metadata
Else if a lot of writes happen. It might happen that an old stat result
is used. Resulting in a wrong file size for the file. For example the
text app when a lot of people edit at the same time.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-05-22 09:21:53 +02:00
Robin Appelman f7153a3d0c
generalize the "lazy folder" concept
makes it easy to use lazy intialized folder in other places

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-05-11 15:36:13 +02:00
Roeland Jago Douma d5850eb28f
Merge pull request #18955 from adrb/swift_upload_large_objects
Large Object support for OpenStack Swift
2020-05-06 13:03:24 +02:00
Robin Appelman 4e6d3d1880
Fix getDirectoryContent implementation for Jail wrapper
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-30 15:27:09 +02:00
Roeland Jago Douma f6b435d0d9
Run the cs fixer for green ci
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-04-30 10:28:44 +02:00
Roeland Jago Douma a1c1b354fc
Merge pull request #18883 from adrb/optimize_swift_14116
Optimize Openstack Swift files download
2020-04-30 09:06:19 +02:00
Christoph Wurst cb057829f7
Update license headers for 19
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-29 11:57:22 +02:00
Adrian Brzezinski e107519295 Large Object support for OpenStack Swift.
Until now, you wouldn't be able to create
objects larger that 5GB.

It's somewhat related with pull #18883

Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
2020-04-27 14:23:59 +02:00
Adrian Brzezinski 354c0996d8 It's a fix for issue #14116,
Improves efficiency when downloading files from Swift storage.
Before, files were downloaded and then pushed back to user.
That behaevior causes all kinds of performance problems.

Now, files are streamed directly to user.

Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
2020-04-27 14:16:50 +02:00
Robin Appelman 436908ee3f
filter out blacklisted files in getDirectoryContent
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-23 13:34:55 +02:00
Robin Appelman 9735b5d61f
Add method to storage backends to get directory content with metadata
Currently you need to use `opendir` and then call `getMetadata` for
every file, which adds overhead because most storage backends already
get the metadata when doing the `opendir`.

While storagebackends can (and do) use caching to relief this problem,
this adds cache invalidation dificulties and only a limited number of
items are generally cached (to prevent memory usage exploding when
scanning large storages)

With this new methods storage backends can use the child metadata they
got from listing the folder to return metadata without having to keep
seperate caches.

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-20 15:45:04 +02:00
Roeland Jago Douma 79cf13e349
Merge pull request #16946 from exussum12/patch-1
Increase max read
2020-04-16 10:37:10 +02:00
Scott Dutton 65e134c13a
Update View.php 2020-04-15 19:10:06 +01:00
Morris Jobke 7fe8477830
Run cs:fix and fix HashWrapper code
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2020-04-15 14:08:56 +02:00
Morris Jobke d8557d0d8a
Merge pull request #19540 from jlutran/fixKeystoneV3Auth
Fix issues with Keystone auth v3 in files_external app
2020-04-15 14:05:32 +02:00
Christoph Wurst 301327135c
Merge pull request #20361 from nextcloud/write-stream-close-on-exception
Close the streams in `writeStream` even when there is an exception
2020-04-15 10:48:24 +02:00
Julien Lutran 3e849daf11
Fix issues with Keystone auth v3 in files_external app
Signed-off-by: Julien Lutran <julien.lutran@corp.ovh.com>
2020-04-15 10:33:08 +02:00
Robin Appelman c2677682c4
Return hashes of uploaded content for dav uploads
hashes are set in "X-Hash-MD5", "X-Hash-SHA1" and "X-Hash-SHA256" headers.

these headers are set for file uploads and the MOVE request at the end of a multipart upload.

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-13 12:57:27 +02:00
Scott Dutton 75b99e85ff
Increase max read
8kb is very low, especially given this will be local files

Signed-off-by: Scott Dutton <scott@exussum.co.uk>
2020-04-12 13:01:29 +01:00
Christoph Wurst 734c62bee0
Format code according to PSR2
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:56:50 +02:00
Christoph Wurst 28f8eb5dba
Add visibility to all constants
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:54:27 +02:00
Christoph Wurst 1584c9ae9c
Add visibility to all methods and position of static keyword
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:51:06 +02:00
Christoph Wurst a7c8d26d31
Add visibility to all properties and move static keyword
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:48:31 +02:00
Christoph Wurst caff1023ea
Format control structures, classes, methods and function
To continue this formatting madness, here's a tiny patch that adds
unified formatting for control structures like if and loops as well as
classes, their methods and anonymous functions. This basically forces
the constructs to start on the same line. This is not exactly what PSR2
wants, but I think we can have a few exceptions with "our" style. The
starting of braces on the same line is pracrically standard for our
code.

This also removes and empty lines from method/function bodies at the
beginning and end.

Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 14:19:56 +02:00
Christoph Wurst 14c996d982
Use elseif instead of else if
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 10:35:09 +02:00
Roeland Jago Douma fb64fda4d8
Merge pull request #20385 from nextcloud/skeeable-http-harden
harden seekable http stream a bit against failures
2020-04-10 08:34:52 +02:00
Christoph Wurst 44577e4345
Remove trailing and in between spaces
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 16:07:47 +02:00
Christoph Wurst 42625a46be
Remove spaces after method or function call
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 16:05:56 +02:00
Robin Appelman 60de74ac40
harden seekable http stream a bit against failures
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-09 14:37:01 +02:00
Christoph Wurst afbd9c4e6e
Unify function spacing to PSR2 recommendation
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 13:54:22 +02:00
Christoph Wurst 41b5e5923a
Use exactly one empty line after the namespace declaration
For PSR2

Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 11:48:10 +02:00
Robin Appelman ccbf3059ba
Close the streams in `writeStream` even when there is an exception
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-08 16:40:56 +02:00
Joas Schilling f8a5812f98
Try to use the display name of file transfers
Signed-off-by: Joas Schilling <coding@schilljs.com>
2020-04-08 15:41:20 +02:00
Roeland Jago Douma f5919d5b83
Merge pull request #20033 from nextcloud/s3-seekable-stream
Enable fseek for files in S3 storage
2020-04-06 21:38:49 +02:00
Julius Härtl 56aa8fd035
Properly hint exceptions thrown by getUserFolder
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2020-04-03 23:06:26 +02:00
Robin Appelman e22a28ecc2
update autoloader
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-03 16:27:20 +02:00
Robin Appelman 3ba46f3b50
add basic tests for s3 seeking and add some error handling if reopen return the wrong range
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 15:23:05 +02:00
Robin Appelman 7b07e7251c
make seekable s3 stream generic
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 15:21:05 +02:00
Lukas Stabe 14401efb0f
Enable fseek for files in S3 storage
Signed-off-by: Lukas Stabe <lukas@stabe.de>
2020-04-01 15:21:01 +02:00
Robin Appelman d9184584e0
restart transaction when cache insert fails due to duplicate
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:17:34 +02:00
Robin Appelman 5199f80196
call cache->insert directly instead of going trough cache->put first when scanning
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:17:28 +02:00
Robin Appelman 9c7da26394
dont try to get existing cache data during scanning if we already know its a new file
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:17:25 +02:00
Robin Appelman 3fb32f6f3d
reuse known parentId when inserting cache entries in scanner
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:14:52 +02:00
Robin Appelman 5439469b68
optimize getMetaData for local storage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:14:48 +02:00
Christoph Wurst 1a9330cd69
Update the license headers for Nextcloud 19
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-31 14:52:54 +02:00
Christoph Wurst 463b388589
Merge pull request #20170 from nextcloud/techdebt/remove-unused-imports
Remove unused imports
2020-03-27 17:14:08 +01:00
Christoph Wurst b80ebc9674
Use the short array syntax, everywhere
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-26 16:34:56 +01:00
Christoph Wurst 74936c49ea
Remove unused imports
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-25 22:08:08 +01:00
Christoph Wurst 23eb0462e5
Use lowercase true, false and null constants
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-25 21:53:04 +01:00
Roeland Jago Douma b300dc5a50
Merge pull request #19782 from nextcloud/quota-wrapper-include-external
Use global used space in quota wrappen when external storage is included
2020-03-24 21:36:01 +01:00
Roeland Jago Douma 3443a00621
Actually check if the owner is not null
Else this can error out on storages that do not have an owner (e.g.
groupfolders).

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-03-16 13:12:10 +01:00
Roeland Jago Douma 9a35e789b4
Get correct mimetype on objectstores
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-03-11 11:14:12 +01:00
Roeland Jago Douma b6245be302
Merge pull request #19746 from nextcloud/locked-exception-forward-existing
pass the existing locks info when making locked exception with absolu…
2020-03-09 19:31:45 +01:00
Robin Appelman c32750ddd9
Use global used space in quota wrappen when external storage is included
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-03-04 18:02:50 +01:00
Robin Appelman fab22ac14c
pass the existing locks info when making locked exception with absolute paths
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-03-02 19:29:44 +01:00
Robin Appelman f972990469
dont try to update storage mtime if we can't get the mtime
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-03-02 15:57:25 +01:00
Robin Appelman 63608ef461
allow writing content directly when creating new SimpleFile
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-02-28 12:55:22 +01:00
Robin Appelman 5ca1929e8c
Create SimpleFile only when writing the content
instead of first creating an empty file and then writing the content.

This solves the overhead of creating an empty file with the common pattern:

```php
$file = $simpleFilder->newFile('foo.txt');
$file->putContent('bar.txt');
```

roughly halving the number of storage and database operations that need to be done when creating a `SimpleFile`.

This is not automatically done with `File` because that has a more complex api which I'm more hesitant to touch.
Instead the `Folder::newFile` api has been extended to accept the content for the new file.

In my local testing, the overhead of first creating an empty file took about 20% of the time for preview generation

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-02-28 12:55:22 +01:00
Joas Schilling dc050fb5e5
Allow to overwrite the path on the cache event
Signed-off-by: Joas Schilling <coding@schilljs.com>
2020-02-13 08:38:33 +01:00
Joas Schilling 08cadf8416
Don't create invalid users
Signed-off-by: Joas Schilling <coding@schilljs.com>
2020-02-12 10:55:27 +01:00
Christoph Wurst 1b46621cd3
Update license headers for 18
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2019-12-20 09:23:25 +01:00
Christoph Wurst b81b824da1
Add typed events for the filesystem/scanner
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2019-12-13 09:41:05 +01:00
Roeland Jago Douma c6e51924c8
Merge pull request #18236 from nextcloud/bugfix/noid/always-detect-mimetype-by-content-in-workflows
Allow to detect mimetype by content
2019-12-12 11:19:55 +01:00
Roeland Jago Douma 4356c91ffd
Fix detection of non extention types
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-12-12 09:58:22 +01:00
Roeland Jago Douma 4fad9e4e6b
Merge pull request #18314 from nextcloud/fixFileIdSearch
Fix ambigious fileID on search
2019-12-10 11:28:40 +01:00
Joas Schilling 107ef72afb
Try more methods
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-12-10 10:56:47 +01:00
tobiasKaminsky 41cb214112
fix https://github.com/nextcloud/server/issues/18313
Signed-off-by: tobiasKaminsky <tobias@kaminsky.me>
2019-12-10 09:47:30 +01:00
Joas Schilling da44c2a414
Some more fixes in detecting the mimetype from the content
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-12-10 09:11:42 +01:00
Joas Schilling 1336dedd5d
Cleanup the code
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-12-10 09:11:40 +01:00
Joas Schilling b92ebb928a
Allow to check for the mimetype by content only
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-12-10 09:10:32 +01:00
Roeland Jago Douma 5dba8d318d
Fix search limit to home
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-12-09 20:38:13 +01:00
Daniel Kesselberg 29575c4d36
Move custom definition logic into method
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2019-12-09 12:10:19 +01:00
Daniel Kesselberg 64aba49461
Ensure that we don't merge broken json.
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2019-12-09 12:09:43 +01:00
Christoph Wurst 5bf3d1bb38
Update license headers
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2019-12-05 15:38:45 +01:00
Roeland Jago Douma 63cb31542d
Merge pull request #17941 from nextcloud/search-by-owner
Allow filtering the search results to the users home storage
2019-12-05 11:04:33 +01:00
Roeland Jago Douma 81b919ed9f
Merge pull request #18163 from nextcloud/fix_file_size
fix filename size
2019-12-04 14:15:07 +01:00
Roeland Jago Douma 76b78edd40
Merge pull request #17833 from nextcloud/propagator-no-negative-sizes
dont set folder size to negative values during propagation
2019-12-04 11:49:26 +01:00
Greta Doci 1110a93fc4 fix filename size, add comment
Signed-off-by: Greta Doci <gretadoci@gmail.com>
2019-12-04 11:04:46 +01:00
Robin Appelman f9e5b33c2b
remove old 'owner' filter
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-12-03 15:20:54 +01:00
Robin Appelman c62637da8b
Allow filtering the search results to the users home storage
This is done by adding a

```xml
<d:eq>
    <d:prop>
        <oc:owner-id/>
    </d:prop>
    <d:literal>$userId</d:literal>
</d:eq>
```

clause to the search query.

Searching by `owner-id` can only be done with the current user id
and the comparison can not be inside a `<d:not>` or `<d:or>` statement

Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-12-03 13:49:37 +01:00
Roeland Jago Douma 6a940d5c74
Merge pull request #18149 from nextcloud/fix/18148/dont-create-bundle-too-early
delay creation of the cert bundle
2019-11-29 09:07:33 +01:00
Arthur Schiwon 0e911d643f
delay creation of the cert bundle
fixes #18148

Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2019-11-28 17:00:55 +01:00
Julius Härtl 9eb1554fad
Properly anotate LockedException in files node api
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2019-11-28 14:17:15 +01:00
Roeland Jago Douma 0568b01267
Improve etag handling
Check if values exist before using them

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-11-27 13:34:42 +01:00
Arthur Schiwon fc16b09a2c
Files first to use own GenericEvent
* those are added to 18 only anyway :)

Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2019-11-26 14:48:59 +01:00
Robin Appelman 74c6beb603
dont set folder size to negative values during propagation
normally this shouldn't be a problem, but cache/storage desync might cause this
so this adds some failsafe to ensure we dont corrupt the cache further

the minimum value is set to -1 instead of 0 in order to triger a background scan
on the folder and figure out the size properly

Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-26 12:05:30 +01:00
Roeland Jago Douma c7d5b8fc49
Merge pull request #18079 from nextcloud/fixes/phpcs
Some php-cs fixes
2019-11-25 14:07:00 +01:00
Roeland Jago Douma 68748d4f85
Some php-cs fixes
* Order the imports
* No leading slash on imports
* Empty line before namespace
* One line per import
* Empty after imports
* Emmpty line at bottom of file

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-11-22 20:52:10 +01:00
Roeland Jago Douma 9a64759af5
Harden watcher
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-11-22 13:38:16 +01:00
Roeland Jago Douma d625d8bd1e
Merge pull request #17824 from nextcloud/fix/dav/catch_forbidden
Catch forbidden http status code
2019-11-20 12:32:00 +01:00
Roeland Jago Douma 8800a7e890
Fix cache moves
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-11-18 12:48:00 +01:00
Roeland Jago Douma c08e803db3
Catch forbidden http status code
If you try to do something on a DAV mount (external or federated share)
that is not allowed. We should not mark the storage as not available but
just fail somewhat gracefully.

Now by catching this and just properly returning the operation will just
fail (and notify the user) which is already a lot better then marking
the storage as unavailable and doing boom.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-11-17 09:11:15 +01:00
Robin Appelman f2724992fc
cast ctime and utime
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-13 12:39:56 +01:00
Robin Appelman 4c97181128
filter out empty filecache extension values
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-13 12:39:52 +01:00
Robin Appelman 066faaec05
fix folder delete
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-13 12:39:46 +01:00
Robin Appelman 43fa746ad9
dont create empty filecache extended rows
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-13 12:39:43 +01:00
Robin Appelman 136c4ef925
dont expose metadata etag for now
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-13 12:39:36 +01:00
Robin Appelman 1b6e647b63
expose filecache extension data in FileInfo
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-13 12:39:32 +01:00
Robin Appelman 842da3f183
store filecache extension fields
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-13 11:30:44 +01:00
Robin Appelman d3b6dbc0bc
move more cache logic to querybuilder
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-11-13 11:30:41 +01:00
Roeland Jago Douma bc9b223e65
Fix DAV mimetype search
Fixes #15048
Catches the case where a full mimetype is sumbitted in the where like
clause. Before we didn't catch this and it was just forwarded as is
causing invalid queries.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-10-28 21:56:29 +01:00
Arthur Schiwon e8095cf737
use OCP\EventDispatcher\Event over Symfony's deprecated Event
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2019-10-17 13:31:48 +02:00
Julius Härtl a9c089064b
Deprecate TemplateManager
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2019-10-16 09:54:17 +02:00
Roeland Jago Douma 5917644536
Merge pull request #17276 from nextcloud/storage-id-eq
Get single storage id using `eq` instead of `in`
2019-10-09 15:17:49 +02:00
Roeland Jago Douma cc6874df19
Merge pull request #17264 from nextcloud/move-from-storage-wrappers
handle moveFromStorage within the same storage even when storage wrap…
2019-09-26 15:48:59 +02:00
Robin Appelman bde791cec9
use eq instead of in for loading single storage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-09-26 15:04:04 +02:00
Robin Appelman 1a8f9b8b1d
log which storage id can't be inserted
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-09-26 15:04:00 +02:00
Roeland Jago Douma 6ac67011f4
Merge pull request #17262 from nextcloud/objectstore-remove-cache-on-delete
dont delete cache entries if deleting an object from object store failed
2019-09-26 11:24:02 +02:00
Robin Appelman 35f317df7b
handle moveFromStorage within the same storage even when storage wrappers are applied to the source storage
the target storage doesn't need additional handling for wrappers as the wrappers implementation of moveFromStorage already deals with that

Any storage based on local storage isn't affected by this as local storage already has it's own way of handling with this

Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-09-25 19:17:06 +02:00
Robin Appelman 733d4b6cca
dont delete cache entries if deleting an object from object store failed
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-09-25 18:09:45 +02:00
Joas Schilling a4ba2113b2
Fix getById for files in appdata and the root mount
In case the path we are currently in is inside the appdata_* folder,
the original getById method does not work, because it can only look inside
the user's mount points. But the user has no mount point for the root storage.

So in that case we directly check the mount of the root if it contains
the id. If it does we check if the path is inside the path we are working
in.

Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-09-17 11:15:59 +02:00
Arthur Schiwon 20901c59d4
emit file events via Dispatcher, too
another step to get rid of hooks and emitters

Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2019-09-09 22:56:04 +02:00
Roeland Jago Douma bf6082e119
Merge pull request #16836 from nextcloud/fix/16724/smb-availability
Fix SMB availability status + higher delay on auth issues
2019-09-09 16:50:42 +02:00
Julius Härtl 64fe9bc287
Return the proper jailed path when requesting the root path
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2019-09-04 13:25:17 +02:00
Arthur Schiwon 43bc31bacb
set a storage availability delay on auth issues to avoid lock out
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2019-08-22 12:40:15 +02:00
Joas Schilling 323642454a
Undefined variable response when server is no nextcloud anymore
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-08-20 12:54:58 +02:00
Roeland Jago Douma d14aed1048
Merge pull request #15946 from martink-p/bugxfix/9792/encrypted-external-webdav
Update Encryption.php
2019-08-15 10:22:50 +02:00
Roeland Jago Douma 650927a822
Properly return an int in the getId function of the cache
fixes #16684

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-08-07 20:38:42 +02:00
Morris Jobke e21f440990
Merge pull request #16502 from nextcloud/bugfix/16474
Check the if we can actually access the storage cache for recent files
2019-07-29 16:59:26 +02:00
Morris Jobke e45fb5fa3e
Fix typo in comment
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2019-07-29 14:55:43 +02:00
Roeland Jago Douma fb78cd3ed8
Merge pull request #16570 from nextcloud/enh/supress_touch_error
Supress warnings touch can generate
2019-07-29 10:39:46 +02:00
Roeland Jago Douma 9ef23e2362
Merge pull request #16558 from nextcloud/enh/less_verbose_locked_logging
Do not log all locked exceptions
2019-07-27 10:39:11 +02:00
Roeland Jago Douma 1cc8a2f5d2
Supress warnings touch can generate
We already catch the result value. Having the warning being logged
explicitly doesn't help and polutes the log.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-07-26 17:26:59 +02:00
Roeland Jago Douma 4cc41cb4c7
Do not log all locked exceptions
This can happen for valid reasons (multiple users writing at the same
time) with for example the text app. Apps should properly handle it. No
reason to log it by default.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-07-26 14:55:13 +02:00
Roeland Jago Douma 0487144b26
Remove deprecated searchByTag
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-07-26 12:29:19 +02:00
Julius Härtl e43b341b04
Add additional check for read permissions
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2019-07-24 14:01:24 +02:00
Julius Härtl 3674f6fa2d
Check the if we can actually access the storage cache for recent files
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2019-07-24 14:01:24 +02:00
Morris Jobke 48653d1a27
Merge pull request #16440 from marcelklehr/fix/objectstorage-put-contents
Fix File#putContents(string) on ObjectStorage
2019-07-17 22:38:41 +02:00
Marcel Klehr d46744e2f1 Fix File#putContents(string) on ObjectStorage
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
2019-07-17 14:58:56 +02:00
Roeland Jago Douma 8ca2b31804
Do not keep searching for recent
If userA has a lot of recent files. But only shares 1 file with userB
(that has no files at all). We could keep searching until we run out of
recent files for userA.

Now assume the inactive userB has 20 incomming shares like that from
different users. getRecent then basically keeps consuming huge amounts
of resources and with each iteration the load on the DB increases
(because of the offset).

This makes sure we do not get more than 3 times the limit we search for
or more than 5 queries.

This means we might miss some recent entries but we should fix that
separatly. This is just to make sure the load on the DB stays sane.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-07-16 19:14:55 +02:00
Roeland Jago Douma e953205908
Use HTTP1.1 to read S3 objects
Some of the READs otherwise use HTTP/1.0 which is not always supported
by all backends. HTTP/1.1 is there since 1999 way longer than S3 so safe
to assume it is always there IMO.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-07-10 11:42:22 +02:00
John Molakvoæ 5a03189ce7
Mimetype list integrity check should not fail if it's changed (#15810)
Mimetype list integrity check should not fail if it's changed
2019-07-07 20:01:58 +02:00
Roeland Jago Douma 5312a07f55
Merge pull request #16107 from nextcloud/local-check-path
verify that paths are valid for recursive local move
2019-06-27 16:36:30 +02:00
Robin Appelman f39605dabc
verify that paths are valid for recursive local move
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-06-27 11:10:08 +02:00
Roeland Jago Douma 2f5810f26e
Merge pull request #15832 from nextcloud/bugfix/noid/fulltext-search-groupfolders
Fix full text search for groupfolders
2019-06-25 21:43:37 +02:00
Morris Jobke 9dac9710e4 Fix full text search for groupfolders
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2019-06-17 12:52:22 -01:00
martink-p 9cff71f5b0 Update Encryption.php
Signed-off-by: martink-p <47943787+martink-p@users.noreply.github.com>
2019-06-17 12:13:29 +02:00
martink-p 8b22ede5f3 Update Encryption.php
Signed-off-by: martink-p <47943787+martink-p@users.noreply.github.com>
2019-06-17 12:13:29 +02:00
martink-p 39a4fe40a3 Apply suggestions from code review
Co-Authored-By: Daniel Kesselberg <mail@danielkesselberg.de>
Signed-off-by: martink-p <47943787+martink-p@users.noreply.github.com>
2019-06-17 12:13:29 +02:00
martink-p 4b9675df6c Update Encryption.php
Signed-off-by: martink-p <47943787+martink-p@users.noreply.github.com>
2019-06-17 12:13:29 +02:00
Robin Appelman b4e2d8f8ad
handle storage exceptions when trying to set mtime
not all storage backends can handle setting the mtime and they might
not always handle that error correctly.

Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-06-06 16:09:27 +02:00
Xheni Myrtaj d227f4d34c
Do not fail integrity check if mimetype list is changed
Signed-off-by: Xheni Myrtaj <myrtajxheni@gmail.com>
2019-05-29 22:43:23 +02:00
Roeland Jago Douma 58fe0b8032
Merge pull request #15776 from nextcloud/get-by-id-shares-last
search files by id in shared storages last
2019-05-29 21:56:15 +02:00
Robin Appelman 98dd526010
search files by id in shared storages last
this prevents unneeded recursion when the file is not in the share

Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-05-28 20:03:45 +02:00
Julius Härtl 67dd4b018a
Check for free space on touch
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2019-05-28 13:06:59 +02:00
Morris Jobke 96e892770d
Merge pull request #15715 from nextcloud/s3-bucket-name
Allow other code to use the s3 client from object store
2019-05-24 15:03:22 +02:00
Robin Appelman 834db3af2c
add method to get bucket name from s3 storage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-05-24 14:02:03 +02:00
Robin Appelman 06a119242e
make ObjectStoreStorage::getURN public
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-05-22 12:17:33 +02:00
Robin Appelman 439b341e89
add method to get the used object store from the storage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-05-21 17:18:00 +02:00
Robin Appelman 6a30c65a32
remove extra param in cache update call
param was leftover from earlier iterations of the "update event"

Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-05-07 14:53:02 +02:00
Arthur Schiwon 009c2dfbd5
take return of mkdir into account, throw exception on failure
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2019-04-29 16:45:59 +02:00
Roeland Jago Douma dedb84acb0
Always use a temp file for swift writes
Apparently the if statement doesn't work in all cases (even if I could
not reproduce it). So for the time being we will just not directly
stream to swift.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-04-18 21:04:27 +02:00
Roeland Jago Douma 5d360bd16f
Harden appdata putcontent
If for whatever reason appdata got into a strange state this will at
least propegate up and not make it do boom the next run.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-04-08 13:22:00 +02:00
Vlastimil Pecinka 1c3abda280
Store user object later in the code
Signed-off-by: Vlastimil Pecinka <pecinka@email.cz>
2019-03-21 21:57:47 +01:00
Robin Appelman cc29f5d719
always allow moving mountpoints inside the same mountpoint
even if that mountpoint isn't normally a valid target for moving mounts into

Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-03-19 15:56:18 +01:00
Roeland Jago Douma 6f88f28a15
Merge pull request #14689 from nextcloud/rename-moveable-mount-readonly-parent
allow renaming the root of a movable mount even if the parent folder …
2019-03-15 16:48:34 +01:00
Joas Schilling 17d5e47937
Remove spaming debug logs
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-03-15 12:13:44 +01:00
Robin Appelman 3ca1e90044
allow renaming the root of a movable mount even if the parent folder is readonly
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-03-14 15:27:25 +01:00
Robin Appelman 631ae17dce
handle long etags from dav external storage
we can only store etags up to 40 characters long in the database, so when we get an etag that's longer we simply hash it to bring down the length

Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-03-14 14:46:39 +01:00
Morris Jobke 772303309d
Merge pull request #14425 from cowai/fix-multiple-incomplete-folders
Do not calculate folder size for parent that also needs proper scan, fixes #3524
2019-03-08 09:12:33 +01:00
Roeland Jago Douma 1b27e9578d
Merge pull request #14424 from nextcloud/fix/13554/swift_to_tmp
Use a tmp file for swift writes
2019-03-06 15:33:15 +01:00
Ari Selseng d16cfb519e Avoid calculating folder size for parent that needs scan.
Signed-off-by: Ari Selseng <ari@selseng.net>
2019-03-06 15:23:37 +01:00
Morris Jobke debd32b461
Merge pull request #14175 from skydiablo/master
Fix ObjectStorage expired token (v3)
2019-03-05 18:59:45 +01:00
Roeland Jago Douma 1153123b90
Use a tmp file for swift writes
Else this leads to a seekable stream error with chunked uploads

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-03-04 13:11:59 +01:00
Julius Härtl c778032276
Properly handle null as ownerId if file system access is denied by access token
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2019-03-01 20:57:01 +01:00
Joas Schilling 01b4db62fb
Add dispatcher events to User and Group objects
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-03-01 20:56:59 +01:00
Roeland Jago Douma df55781814
Merge pull request #14254 from sparrowjack63/master
Dont check Bucket Name in Nextcloud
2019-03-01 13:46:04 +01:00
S. Cat e3142a91b2 Bucket name - Alert in debug mode
Alert in debug mode only when trying to mount non compatible DNS bucket name (in order to not flood the logs)

Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +01:00
S. Cat 381ae44d38 Correct code indent
Just correct code indent.

Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +01:00
S. Cat 77473b0228 Warning on S3 not compatible DNS bucket name
The new created bucket should respect the DNS compatibility, nevertheless, Nextcloud should accept to mount "old created" buckets that does not respect DNS compatibility (Backward compatibility, or compatibility with CEPH).

Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +01:00
Sebastien Cat 59c3d8d85f Dont check Bucket Name in Nextcloud
Signed-off-by: Sebastien Cat <sparrow.jack63@gmail.com>
Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +01:00
Robin Appelman 8fc47c6f00
add option to get raw size (without submounts) from fileinfo
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-02-27 15:35:44 +01:00
Morris Jobke 50d7f87eab
Merge pull request #14269 from nextcloud/cache-update-event
add event for cache update
2019-02-19 11:07:59 +01:00
Robin Appelman 092f78d7a4
add event for cache update
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-02-18 16:50:41 +01:00
Roeland Jago Douma f6f002e2ad
Merge pull request #14210 from nextcloud/fix/14192/fix_empty_uploads
Fix empty file uploads to S3 (and other streaming storages)
2019-02-18 15:39:17 +01:00
Roeland Jago Douma 104616c1da
Use CountWrapper instead of almost identical CountReadStream
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-02-15 13:24:58 +01:00
Roeland Jago Douma 61d66d7954
Fix empty file uploads to S3
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-02-15 08:41:39 +01:00
Roeland Jago Douma 62ddb124a1
Filter out versions and trashbin
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-02-13 21:27:25 +01:00
Roeland Jago Douma cb67c6aa00
Loop for the recent search
Else it might happen that less entries than requested are returned

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-02-13 21:22:16 +01:00
Roeland Jago Douma 4c7424d38b
We need to initialize the storage else the jailed path is always null
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-02-13 21:01:17 +01:00
Volker d06c25b9c8
Update SwiftFactory.php
Auth-Service (in explizit v3) will recheck the cached-token and will end in an "token expired exception".
2019-02-13 12:26:27 +01:00
Morris Jobke baa6a2d52c
Merge pull request #13748 from nextcloud/cache-insert-event
add event for inserting cache entries
2019-02-07 16:35:02 +01:00
Robin Appelman 748bcd407b
add event for inserting cache entries
this provides a reliable way for apps to listen to new files
without the need to of cache wrappers to hook into inserts themselves
(something which isn't 100% reliable)

Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-02-07 15:51:30 +01:00
William Pain c26bdd4731 Fix: Check if `$this->params['user']` is an array
Signed-off-by: William Pain <pain.william@gmail.com>
2019-02-05 09:56:42 +01:00
Robin Appelman 1d322d3b37
always use multipart uploader for s3 uploads
the multipart uploader handles non seekable streams while `upload` does not

Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-01-28 14:51:19 +01:00
Morris Jobke 90e985fe4f
Merge pull request #13739 from nextcloud/cache-cleanup-change
cleanup shared lock if changing to exclusive lock failed
2019-01-22 16:11:46 +01:00
Robin Appelman f69c2d1e84
cleanup shared lock if changing to exclusive lock failed
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-01-22 15:32:48 +01:00
Morris Jobke c3b22f05bb
Merge pull request #13468 from rummatee/issue13276
don't close input stream when writing in encrypted file
2019-01-14 16:21:08 +01:00
Morris Jobke 5a27e54f4b
Merge pull request #13032 from nextcloud/objectstore-write-exists
upload new files in objectstore to a .part path first
2019-01-14 11:26:29 +01:00
Robin Appelman 615fb8cd77
Cache tokens when using swift's v2 authentication
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-01-13 19:59:48 +01:00
Florian Schunk c2f8df0573 don't close input stream when writing in encrypted file
Signed-off-by: Florian Schunk <florian.schunk@rwth-aachen.de>
2019-01-10 00:25:07 +01:00
Roeland Jago Douma d0956c9a42
Followup 12833, gracefully handle the getting of /
Else this breaks the app page

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2018-12-21 11:08:19 +01:00
Joas Schilling 6788e6e75c
Merge pull request #12883 from nextcloud/appdata-performance
try to grab the appdata folder directly without going trough the whole tree
2018-12-17 15:11:28 +01:00
Robin Appelman d6bf5d4384
upload new files in objectstore to a .part path first
This prevent the object store and cache from getting out of sync
when an objectstore silently fails or the php process get's killed
during the upload without giving us the chance to cleanup

Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-12-12 15:24:40 +01:00
Robin Appelman 6c9f2644cf
Add objectExists to objectstore interface
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-12-12 14:17:19 +01:00
Robin Appelman 740ea0d30e
try to grab the appdata folder directly without going trough the whole tree
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-12-06 15:10:08 +01:00
Morris Jobke 60681decdd
Merge pull request #12865 from nextcloud/fix/do_not_propogate_in_appdata
No need to propogate changes in appdata
2018-12-06 08:50:22 +01:00
Morris Jobke ecc8068e66
Merge pull request #12857 from nextcloud/fix/emit_write_update_once_on_touch
Emit write update only once on touch
2018-12-05 22:54:47 +01:00